Niklas SOC progress and thoughts

End of GSoC

niklasbe — Tue, 19 Aug 2008 23:20:08 +0000

Yesterday(august 18th) was the code handin deadline for GSoC. I’ve enjoyed being part of the MusicBrainz community. I hope that you in the near future get to enjoy the outcome of this project

I’ve been in contact with many friendly MusicBrainz people. Thank you all for commenting, helping out with coding questions, mb_server questions, keeping me company on IRC late nights etc etc. Thanks to Robert Kaye who mentored me, and Kuno for coding a collection MBId submitting tool.

Things have gone quite well. The basic functionality plus some is finished. It still has to be more carefully tested and there is room for improvements. I’m continuing the work, and Robert will set up a server soon for public testing.

Edit: Oh of course… Thanks Google!

What is done and what needs to be done

niklasbe — Mon, 04 Aug 2008 01:56:34 +0000

Time is closing up on the final code submission deadline when the project shall be done. Things has gone pretty ok.

This deadline is the 18th of August. On the 17th I will be leaving to visit my girlfriend and the plan is that while I’m gone people can try out the collection functionality to find bugs and to comment on it. Then I will be back and fix bugs.

What is done so far is:

Web service for inserting and removing MBIds from collection
Displaying user’s collection
Displaying missing releases
Users can select artists to view missing releases of, and to be notified about new releases.
Users can remove those mentioned in the above bullet point
Preference whether to send notifications by e-mail or not
Preference for how many days in advance the user want to be notified about new releases

What is not done:

Displaying new releases on the website

Other things to do:

Ignore versions of releases when displaying missing releases
Optimize code
Improve collection web service response format
Bug fixing

What is mentioned above is “must have” functionality and slightly more. That something is not listened to not necessarily mean it wont be implemented. The tool for submitting MBIds has not been listed as his kindness Kuno(warp) is working on that.

The “other things to do” list is a bit short. I’ve got a paper with notes of small fixes that will be done but I omitted details here.

“weekly” progress report

niklasbe — Tue, 08 Jul 2008 16:28:18 +0000

Ive been doing progress slowly this week. Have been working with Mason, integrating things into the website. Thats going quite well. Right now Im doing preferences and hope to finish that today. Thats roughly what I’ve done.

Now I’ll try to finish the preferences page today. Once that is done I’ll do the missing releases page, and add support for the preferences in the collection page. The one that displays all releases in collection.

Wiki pages

niklasbe — Thu, 03 Jul 2008 00:20:54 +0000

I have received feedback asking for more detailed info about the project and the data model. So I created two wiki pages for this. Please have a look and if you feel there’s something missing let me know!

You find them at http://wiki.musicbrainz.org/MusicCollection. It is linking to the other page - http://wiki.musicbrainz.org/MusicCollectionDataModel.

What I have been doing and what Im working on now

niklasbe — Sun, 29 Jun 2008 20:06:12 +0000

I usually find it hard to keep focus and get things done. To put more pressure on me I’ll be summarizing what I’ve been working on, and what I’m currently working on. I will try to make these posts quite often, at least weekly. And you guys can also help by telling me that Im too slow in the comments

So what have I been doing lately? Not very much, but I would say I know the parts of mb_server that Im using quite well now. I have almost done the webservice for adding and removing releases in one’s collection. Got user authentication left, and some details.

I have started working on a class which grabs collection data from the database. Thats working quite well. It can grab artists in the collection, releases in the collection and missing releases. Id’s of artists to view missing releases of and id of the collection have to be entered into the tables manually right now.

So what I will be doing now is to integrate this into the website – preferences and displaying collection info. I’ll do upcoming releases later probably.

Have to look into Mason, but I’ve been told its easy to use so that shouldn’t be a problem.

I have not produced very much, but I feel that Im past the period of chaos when everything was new to me and I did not know how things worked. So Im satisfied with the results so far. I believe Im on schedule.

Where to put files, and which files are modified

niklasbe — Sun, 29 Jun 2008 01:24:17 +0000

It has been quite a while since the last post. Tomorrow I will write a post which summarises what I have done, and what Im going to do next. That summary post will be recurring at least weekly starting tomorrow.

In this post I will list new files created by this project, and files it will modify. Im suggesting some paths for these files. Questions, comments, ideas, anything is welcome. There’s many people out there who know mb_server way better than I do.

I’ll just dump a list on you… There are comments for some of the files. These are the new files:

/htdocs/show/collection/index.html – display overview of collection. also display notifications about new releases
/htdocs/show/collection/missing.html – display missing releases

/htdocs/user/collectionpreferences.html – preferences…

/cgi-bin/MusicBrainz/Server/CollectionInfo.pm – logic for what is in the collection and what is missing
/cgi-bin/MusicBrainz/Server/WatchArtists.pm – logic for watching artists, website notifications about new releases
/cgi-bin/MusicBrainz/Server/Collection.pm – back-end for the webservice. adding and removing releases from collection
/cgi-bin/CollectionPreference.pm – back-end for collection preferences

/cgi-bin/MusicBrainz/Server/Handlers/WS/1/Collection.pm – webservice for adding/removing releases in collection

/admin/SendReleaseNotifications.pl – script sending e-mail notifications about new releases. run daily(hence modifying daily.sh)

/admin/sql/updates/20080711.sql — update script for the raw database

/htdocs/comp/collection – Mason component for displaying collection
/htdocs/comp/collectionmissing – Mason component for displaying missing releases

And heres files Im modifying:

/cgi-bin/UserPreference.pm – logic for settings

/admin/vh_httpd.conf
/admin/startup.pl

/admin/cron/daily.sh

/admin/sql/vertical/rawdata/CreateIndexes.sql
/admin/sql/vertical/rawdata/CreatePrimaryKeys.sql
/admin/sql/vertical/rawdata/CreateTables.sql

/admin/sql/vertical/rawdata/DropIndexes.sql
/admin/sql/vertical/rawdata/DropPrimaryKeys.sql
/admin/sql/vertical/rawdata/DropTables.sql

/admin/SetSequences.pl
/admin/ExportAllTables
/admin/MBImport.pl

/htdocs/comp/layout/topmenu – adding links to pages

/htdocs/comp/artisttitle – adding links for “watching” and selecting to “display missing releases of” an artist

Thats it for this post.

Edit: snuck in /htdocs/comp/collection and /htdocs/comp/collectionmissing in the list of new files.

Database design and a question to users

niklasbe — Tue, 17 Jun 2008 23:18:39 +0000

Things have been slowly moving forward. I’ve been away for 5 days, visiting friends and family. Got sick when I came back, so havent done much since then either. Hopefully I will feel better tomorrow.

Anyways me and Robert Kaye have been working on updating the data model I created for my project proposal. One question arose… Should one be able to separately specify artists to watch for new releases of and artists to view missing releases of? Of course it is nice to be able to specify more, but it would only be annoying if one always want to both “watch” and display missing releases of same artists. What do YOU think? Please leave a comment! And also… Do you want to be able to specify release types(official, bootleg, EP, single etc…) to ignore separately for “watching” and viewing missing releases? Or do you prefer same setting for both?

(Click the image to view it in its original size)

This data model facilitates the “must have” features listed for my project(functionality.rtf) and some of the other functionality as well. More attributes and possibly tables can be added later if time allows me to implement more functionality.

I will put together a more extensive description of the data model for the wiki later when it has been more finalized but until then… First of all, collection_info is the most interesting table. I think you can figure out most attributes. ignorecollectionattributes and ignorewatchattributes are lists of the release types to ignore, the ones to ignore when “watching” for releases and those to ignore when displaying missing releases in the user’s collection.

Oh yes and by the way, “join tables” are blue, the other music collection related tables are green and the already existing tables that the music collection make use of are in the orange rectangle.

collection_watch_artist_join is a join table used for knowing which artists the user want to watch for new releases of, while collection_discography_artist_join knows about which artists the user want to see missing releases of.

collection_ignore_release_join store releases that the user want to ignore “missing status” of – so that the website wont display those releases as missing even though they might be. And collection_has_release_join knows about which releases the user have in his/her collection.

As usual, please give positive/negative feedback! The latter is even more appreciated, though I also appreciate positive feedback

Updated “data file format”

niklasbe — Wed, 04 Jun 2008 22:31:16 +0000

“Data file format” is no longer a data file. Thanks everyone for the comments. As ijabz pointed out its an unnecessary step to create a data file instead of separating data using multiple attributes.

So… to add tracks you send a comma separated list with the attribute name addtracks. The attribute name removetracks is used for removing. addalbum and removealbum is used for manipulating the album data.

This should be sent as POST data to something like /mc/1/collection/niklas where mc would be some name for the service, 1 is the version of the service, collection because it is a collection and niklas would be the user whoms collection we are updating.

Tomorrow morning(actually its this morning) Im having my final exam, after that I hope I will get lots of more things done.

Data file format and response format

niklasbe — Sat, 24 May 2008 18:39:02 +0000

I have been thinking some about the format of the data file, and the format of the response.

To make the service resemble XMLWebservice Im thinking of dividing the URIs to where the POST data is sent(Thanks Kuno). One for adding, one for removing. And also it might be a good idea to also separate “tracklist” and “albumlist” to different URIs. Feedback please

My idea right now of the data file format is like this:


version:version name goes here
begin public collection name of collection here

begin tracklist

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

end tracklist
begin albumlist

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

end albumlist

end collection name of collection here

Where those series of X represent UUIDs of a MBID. There is two types of lists – albumlist and tracklist. Tracklist contains UUIDs for tracks, albumlist UUIDs for albums(releases).

I named it album instead of release, because I believe they will be named albums in NGS?

This format is supposed to facilitate all features, including the “nice to have” ones.
Is there some other format used in Musicbrainz? Something that I should make this format resemble?

I also made a rough sketch on the response format. See following:



      
          
               123
               456
               
                    Error message as string here
               
          
          
               12
               34
               
                    Error message as string here

The “details” name is bad in my opinion. Could not come up with anything better. Will revise it.
Both addcount and removecount should not be in the same response though. Theyre just both there to illustrate that they are available.

Im going to make another blog post soon about other updates and comments.

MusicBrainz summit in London

niklasbe — Mon, 12 May 2008 01:09:29 +0000

Came back from London earlier today. Had a great weekend. Thanks to all the MusicBrainz people, and the others who participated. It was very interesting. Nice to get to meet many of the people I have been and will be in contact with. It was interesting to get to know what is going on with MusicBrainz. Learned more about the organization.

And I received good suggestions and feedback on my summer of code project. I have added these suggestions to a file Im keeping with the features of my project. This file can be found here. All of the suggestions have been added as the type of functionality that I would like to implement if there is time. Will keep these in mind when developing, but focusing on the basic functionality which is quite much. I think I should prioritize the extra functionality when I have implemented the basic functionality.

People also pointed out some things that I have to think about – series of albums(disc 1, disc 2 etc), various artists and duplicates… As I understood it people was wondering if for example you have one complete disc of a two disc album – will the other disc be flagged as missing? I think it should… Comments please Or perhaps the issue is that multi disc album are not “connected” in the MusicBrainz database(?)? Various artists might be a problem.

Im not sure whether my implementation should use NGS or not? I dont want to go deeper into various artists right now in this blog post, as I know too little about details of NGS and the current schema.

Duplicates… How should various versions of releases be dealt with? E.g. when an album has a UK release, USA release etc… Someone suggested using the earliest release. Seems like a good solution.

Since the last post I have been playing a bit with LibTag. I think this will be the library I will use for extracting MBID from music files. By the way I understand that python would be the by the community preferred language for the tool. Too bad I dont know python, because learning two new languages and then implement this project using two new languages might take too long. Took me a while to understand the TXXX id3 tag by the way. But I got it now… At first I was trying to figure out what those X was supposed to be replaced with

Thats it for this post. Perhaps its a bit of a mess. Wanted to put down some thoughts. Hey heres a picture of me, the other MusicBrainz SoC students, two(of three) mentors – Robert Kaye and Philipp Wolfer: http://flickr.com/photos/mayhem/2482047071/

This picture was taken in the ball pit at the last.fm office where the MusicBrainz summit was held.

Oh yes.. I might have misinterpreted something, or forgot about something. If so please tell me. And comments are very much appreciated!