Text to Soundtrack

During my second year of Creative Technology I, together with one other Create student and two Computer Science students, created an application which would search for a fitting song that would play alongside the text of a book. The application used a bag of words algorithm to figure out what category the piece of text belonged to. Nine different categories were defined, one for every ‘mood’, with each their set of words which would belong to the particular mood. The (Java) application loaded in an EPUB file (commonly file format used for e-books) and would display one or two paragraphs on the screen for the reader to read.

From the paragraph, the algorithm would calculate a probability of the paragraph to have a certain theme based on the percentage of words that would fit in the designated category. Then the category with the highest percentage would be chosen. An appropriate song was then chosen by using two web APIs. One API provided additional meta information for Spotify songs, which we needed to find the ‘wordiness’ of a particular song. As, through user testing, we found that user did not like spoken or song text in their background music. Then the Spotify API was called to get the data of the song to be played.

For the user test we used a fake prototype where we would play music according to were in the text the user would be. So called ‘Wizard of Oz’ testing. The results from the test was that half of the participants liked to have tailored music played during reading, and half did not like music at all. The music that was chosen was found fitting by all participants. We also experimented with sound effects during reading, but this was almost unanimously disliked. Therefor this feature was dropped. When the prototype was finished and working, we did two additional usability tests in order to see if we had succeeded in creating a soundtrack chooser that would chose appropriate music during reading. More than tree quarter of all users liked the feature.

My role during this project was to create the user interface of the Java application and to implement the EPUB reader functionality and the bag of words classifier. Communication with the APIs, documentation of the project, audio and the file IO was handled by the other members of the team.