The never-ending quest to rate and organize my iTunes library - part 2 of 2

In part1 of this blog post, I indicated that I had been rating the music in my iTunes library for about 8 years. Having thought about this some more, I think its probably been at least a decade. But after all of that time, I am finally finished! For the last four years I've kept track of how many songs remained in my 'To Rate' playlist, and so I can graphically show you how I finished this very long marathon.

Here is an explanation for the various milestones along the way to having no unrated songs remaining in my library.

a) January 2009 At this point I was only rating music that I had purchased (CDs, iTunes, and Amazon) and had just over 1,500 songs left.

b) August 2009 I tend to buy CDs in splurges rather than regular purchases every week or so. This jump probably represented about 10 new CDs. Most of the small spikes on this graph represent new music purchases.

c) February 2010 Finished! All of my purchased music had been rated...but what about all of the other music in my library which I had legally acquired for free. How much could there be?

d) February 2010 Well quite a lot as it turns out. Since 2005, the South by Southwest festival (SXSW) has provided free torrents of music by people performing at SXSW (one song per artist). If you didn't know, a lot of artists play at SXSW and these torrents are big. You can still get hold of all torrents from this site. The current running total is about 9,000 songs (45 GB!) and every year the torrents seem to get bigger.

These torrents are usually provided as two files, with the second one being smaller (and released a little bit after the first file). So this big spike represents me starting to rate 5 and a half years of SXSW music. Sidenote: a shout out to Danny Novo who had made some cool SXSW iTunes album artwork every year that the torrents have been available.

e) March 2010 I think this smaller spike represented the second of the two 2010 SXSW torrent files, with the majority of the SXSW music included in the previous huge spike.

f) and g) February & March 2011 The two SXSW torrents for 2011. You may note that initially I seem to rate the music very quickly and then it slows. This is because I tend to sort the yearly SXSW torrent by song length. Songs that are extremely short (< 1 minute) or extremely long (>7 minutes) tend to be music that I don't like and I can quickly process these. I have also found that songs that feature 'death' in the song title, or 'black' in the artist name are also not to my liking. Finally, I am not a fan of rap music so artists with 'MC' or 'DJ' in their name can also be quickly filtered.

h) March 2012 SXSW 2012.

i) August 2012 A bunch of new CD purchases.

j) January 2013 I think I had finished all of the SXSW music at this point, but then discovered a whole bunch of previously purchased music that had slipped through the cracks (pretty much of all of my classical music), plus I also had another bunch of new CDs to rate.

k) March 2013 SXSW 2013.

l) April 2013 Each year I get quicker at doing the initial pass of that year's SXSW torrent, and I think you can see how quickly I go through the long/short/black/death/rap songs (see steps f) and g) above).

m) August 2013 Finished. Done. The End. No more rating. right?

What now?

The SXSW torrents represent a real mixed bag of music. If you drove coast-to-coast and kept on randomly flipping through radio stations — preferably college radio stations, there is not much mainstream pop music at SXSW — it would get close to the experience of listening to a SXSW torrent. So unlike the music that I buy, where I have pre-selected albums based on one or more songs that I have heard and that I like, the SXSW music provides no such guarantees.

What tends to happen is that I discover a bunch of new artists through songs that I really love. However, a large swathe of SXSW music is either not to my taste, or is just a bit 'meh'. In fact, I realized that I had ended up with about 1,500 2-star rated songs from my SXSW torrents. While having some 2-star songs in the middle of a CD that own is one thing, these 2-star SXSW songs represent a potential waste of disk space.

As I am a iTunes Match user, I deleted all of these songs from my iTunes library (after ensuring that they had first been matched by iTunes). This frees up some disk space but means I can still play these songs and download them again if I really want them.

However, it still irks me a little to know that these 2-star songs are part of my library, even if they are not taking up any space. So because I am a masochist, and because I can't abide to delete them all in case there are some gems buried amongst them, I decided to re-rate all of these songs. If I decide that I really do like a song, I upgrade it to a 3-star rating. Otherwise I permanently delete it (removing it from iCloud). So far, I've processed about 200 songs and it will probably take me several more months before I finish this.

And then I am done.

Really done.

Until the next SXSW!

The never-ending quest to rate and organize my iTunes library - part 1 of 2

This is a blog post that I've been intending to write for many, many years. The final impetus for getting on with it was seeing David Sparks write about how he rates his music in iTunes. Before I reveal the minutiae of how I rate my own music — a system which is very similar to David's — I should give some background to my music collection and my long attempt at organizing it.

My music

I consider that I own a fair amount of music. I think that the new standard to be used when deciding if you own a lot of music, is whether you find iTunes Match's 25,000 song limit restrictive. Currently, I have 13,275 items in my iTunes library. These include:

• 9,715 songs ripped from CDs I own
• 2,209 songs from free SXSW downloads (more of this in part 2)
• 840 songs purchased from iTunes
• 121 songs purchased from Amazon MP3 store
• 70 songs that were available as free downloads (predominantly from artists websites)
• 35 self-recorded songs or musical sketches
• 23 audiobooks
• 22 podcasts (I mostly use Downcast to manage podcasts)
• 2 E-books (I mostly buy Kindle books)

I can be so precise about this breakdown because I use a set of custom 'tags' for nearly all items in my iTunes library. These tags simply consist of adding text to the 'Comments' field of tracks; I use tags such as [CD], [ITUNES], [AMAZON] etc. The addition of square brackets makes it very unlikely that this text will exist anywhere else in a comments field. It also makes it easy to include as part of a Smart playlist. I'm curious whether the tagging feature of OS X Mavericks will allow tagging of songs in iTunes, which might mean I can replace these comments.

An alternative way of categorizing my music is to break it down by decade of recording. I try to change the dates on tracks to the original recording dates rather than the date that the CD/album was issued. This is mostly a problem for older music that only exists on iTunes from compilation albums or re-issues.

I think this represents a good spread across the decades (maybe Jazz fans would beg to differ). Another way of looking at this is that 11% of the music that I own was recorded before I was born (1971).

Rating songs

I've been using iTunes ever since I purchased my first Mac in 2001 (the era of rip, mix, and burn). I started ripping CDs from around this time, slowly working my way through my CD collection. At some point (maybe around 2005) I started to systematically rate my songs as follows:

1 Star - 2.8% of my library

Things I never want to hear. This sometimes include songs that I really don't enjoy on albums that I otherwise like (typical culprits being low-quality live tracks, or extremely long and unfocused instrumental jams). However, it mostly includes things like silent tracks that are used as interludes on some albums, or spoken tracks (think of all the classic radio DJ interludes on the Reservoir Dogs soundtrack). Giving all of these tracks 1 star means I can create smart playlists that will never randomly shuffle between music and spoken word (or silence).

2 Stars - 44.2% of my library

Songs that I am largely indifferent to. On a good day I might be in the mood for a 2 Star song, but on a bad day I might be offended by one. This category represents a large slice of my iTunes library, reflecting my opinion that there are many great songs which exist on largely mediocre albums (and I like to buy albums rather than individual songs). Perhaps my tendency to down-rate rather than up-rate songs is reflected by the fact that out of about 750 full-length albums, I have only 33 with an average rating greater than 3.5 stars (I have a Perl script that generates average ratings).

3 Stars - 44.2% of my library

Currently, I have almost an identical number of 2 star and 3 star songs. A 3 star song must have at least one great quality about it. This could be the lyrics, the melody, or even the performance of a single instrument. Often it is just a good hook in part of the song, or just the energy that the song imparts. I pretty much always enjoy listening to a 3 star song, but I wouldn't want to listen to the same song over and over again.

4 Stars - 7.9% of my library

There is a big drop in the number of songs that make it to this level. A 4 star song must be amazing. It must have at least two or three great qualities about it and I will almost never tire of hearing a 4 star song. The things that stop them becoming a 5 star song could be something minor, but something that might seep through into my consciousness when I listen to it (a song that outstays its welcome, a bad note in one part that can be jarring, or a single line with a flawed lyric).

5 Stars - 1.2% of my library

These songs represent songs that I find exceptional in almost every way. Usually I find them to have a stunning combination of lyrics and music, with fantastic vocal and instrumental performances. Furthermore I find these songs to have superb production values, with great arrangements and pristine sound (letting you hear details of every instrument on the record). In nearly all cases, I find these songs to be flawless. There is nothing about them that I could imagine being improved. It perhaps reflects my overall preferences for what I might broadly define as 'classic pop/rock' that 90% of these songs are less than 5 minutes long. My iTunes library does comprise genres such as Jazz, Classical, and Prog Rock, but they reflect a small proportion of the music that I own.

Finally, I should add that I will often listen to a song 10 times or more before making my mind up as to what the most suitable star rating should be. This, coupled with the tendency to keep on buying new music, is the major reason that it has taken me almost a decade to finish rating my entire iTunes library. My progress in getting there will be explored in part 2, along with a few comments on the challenges provided by the free SXSW downloads, and the question that now looms large on my horizon:

Once you have finished rating the entirety of your 13,275 song iTunes library, what do you do next?

Not So Breaking News

A new idea for some blog posts to single out some examples of 'Breaking News' services that bring us 'news' which is barely worth calling news, let alone breaking news. First offender is from the Breaking News twitter account which today gave us this:

Report: Apple is evaluating plans to offer iPhones with screens up to 6 inches - @WSJ http://t.co/reVDoFcVMg

— Breaking News (@BreakingNews) September 5, 2013

So Apple is 'evaluating' plans to offer iPhones with screens up to 6 inches. When you click through to the Wall Street Journal story, it reveals the source of this 'breaking news' as people familiar with the matter.

I have no doubt that that might be true, but only in as much that Apple is probably also 'evaluating' iPhones that are thinner, fatter, lighter, and heavier. For all we know, Apple might be 'evaluating' iPhones that come with a free pony as well.

In any case, it is vague speculation from an unnamed source about a possible plan that, even if it happens, would almost certainly not take place until a year from now. Whatever this story is, it hardly seems to constitute 'breaking news'.

Remembering my father

As of today, I have passed a sad, and unwanted, milestone. At the age of 41 years, 9 months, and 26 days, I have become older than my father was when he died.

I was eleven when he passed away, just a month shy of my twelfth birthday. However, I was probably only ten when he was first diagnosed with cancer, a brain tumour. In many ways, I think we lost the person we knew and loved several months before he actually died. Brain tumours can cause so many different symptoms depending on their location. Increased headaches. Confusion. Memory loss. These were all symptoms which affected him in the early stages...but even then he was still our dad. He could still function on his own, he still knew who we were, and he could still hold conversations with us. This would all change.

In the summer before he passed away, it was decided that it would be better for me to spend some time away from everything. My two elder brothers — three and six years older than me — were better equipped to deal with what was happening, and remained at home. I spent that summer break from school — what seemed like an eternity when you are eleven — staying with two of my aunts. It was during this time that I was told that my father was not going to make it. I can't remember whether I had already reached this conclusion for myself. Probably not.

Returning home I was shocked to see how much his condition had deteriorated. He could no longer speak, but what hit me the hardest was that he no longer seemed to recognize anyone. I don't think there is anything that could have ever prepared me for that day. To an eleven year old boy, it made no sense at all. He was my dad but he no longer knew me. As summer drew to a close, so did my dad's fight against the cancer. Despite the immediate sadness at losing him, I think we were all glad that he was no longer suffering.

On the one hand, I am grateful that my memories of my father are all good ones. It's possible that I could be looking at the past through rose-tinted glasses, but I don't think so. I can't recall him ever being angry with us, or disappointing us in any way. What more could you ask for? But at the same time these memories are the memories of a child. I'm jealous of my brothers. Not just because they have more years of memories, but because those memories were from a time when they were able to not just talk with him, but converse and more fully interact with him.

As today's date has approached, I have been surprised by how many feelings it has stirred up from the deep. It has made me reevaluate many aspects of my life. I have only just become a father — of a wonderful 5 month old boy — yet my dad had already helped raise three kids by the time he was my age. It has made me think about the preciousness of life and about how we only have so much time on this planet to do the things we want to do.

I'm sad when I think that my dad never got to see how we all turned out — I like to think that he'd be happy with what we have all made of our lives. I'm sad that he never got to see his four grandchildren, and that they will only have passed on memories by which to know him. I'm sad that I never got to discuss 'grown up' things with him, such as talking about politics or even football (he was a huge Arsenal fan). But mostly I'm sad because I miss him.

Love you dad.

My father and me, 1972

Fruit of the Month Foolery inspired by the USDA

According to the United States Department of Agriculture (USDA), August is Get Acquainted with Kiwifruit Month. This amuses me for the implication that the USDA doesn't necessarily want you to eat any Kiwifruit...just become 'acquainted' with them.

This makes me wonder what other fruit-related months the USDA will endorse in future:

• September: Become Vaguely Familiar with Mango Month
• October: Stroke a Strawberry Month
• November: Make Eye Contact with an Elderberry Month

Fly me to the moon...or maybe just take a train to Birmingham instead

The UK government has been planning a new high-speed rail network known as HS2, the first phase of which will connect London to Birmingham (the UK's second city).

I was shocked to discover that the planned timetable for this initial phase spans nine years (2017–2026). The surprise is because the distance for the London to Birmingham route is only 119 miles. Less surprising — but sadly more predictable — is the recent news that the planned costs for the first part of this project are now projected to be 30% above the initial budget (up to £21.4 billion, ~$32.1 US billion).

This seems a lot of money to connect two cities that are not so far apart. I'm sure that this budget has to cover lots of other things besides the actual construction (e.g. purchasing land). But even setting the cost aside, will it really take almost a decade to construct the line? This is the best-case scenario of course, and pessimists among us no doubt imagine that there will almost certainly be delays (assuming the project is not derailed — metaphorically, not literally — by opposition campaigns).

We are not talking about building a railway line across mountainous terrain or having to circumvent huge lakes. However, any route built in the UK does have to face the twin forces of nature that are leaves on the line and the wrong type of snow.

How did things get so bad? After all, we are the country that introduced the first public passenger railway (1825), the world's first intercity railway (1830), and we also have the world's oldest continuously working public railway (since 1758). We used to be good at building railways!

So just to summarize the costs, timetable, and distance of this rail route:

• $32.1 billion
• 9 years
• 119 miles

If we convert this into a cost-per-year-per-mile, we get a figure of $29,971,989. How does this compare to some other public transportation projects? Well, the new Eastern span of the San Francisco-Oakland Bay Bridge will hopefully open this year. It is running over time and budget, and still keeps running into problems. How does this project compare to the UK's HS2 line?

• $6.3 billion
• 11 years
• 2.2 miles

Cost-per-year-per-mile: $260,330,579 (almost an order of magnitude worse than HS2).

One final comparison. Between 1969 and 1972, the Apollo program put 12 men on the moon. This is, as you might imagine, much harder to work out the total cost of this massive project. There are some estimates that try to allow for inflation, and these put the final bill of the Apollo program between $153 and $422 billion. Let's go with the upper limit of this estimate. That gives us:

• $422 billion
• 13 years
• 239,000 miles

Cost-per-year-per-mile: $135,822 (a bargain!).

Of course these very fickle calculations make no effort to factor in how many people make use of these routes. The Apollo missions only took a grand total of 24 people to the moon (and of course not all of them got there), whereas the Bay Bridge carries 280,000 vehicles a day. However, I still can't get past the fact — and again this assumes that the project won't overrun at all — that the 1st phase of HS2 will see an average of just over 1 mile of the route completed per month of the nine year construction phase. British Rail — the former nationalized rail company that ran the UK's railways — once had a series of TV adverts that ended with the proclamation "We're getting there". A modern day update to this slogan might be "We're getting there...slowly".