Drawing Rhythm from the Algorithm

Dr Hu Xiao has a unique understanding of the power of music. For more than 15 years, she has used data to unleash new possibilities from music. Two groundbreaking models she developed automatically recognise the mood of music and have received hundreds of citations and inspired multiple lines of research around the world. Now, she is investigating whether music can be used to enhance learning. 

“Music can do a lot of things in people’s lives and we are trying to expand that field, not only just in terms of recommendations or algorithms, but in trying to see how it can be applied in different areas,” she said.

Her interest in music information retrieval began while she was a postgraduate student at the University of Illinois. An early study she did looked at how people use music for different activities and occasions, such as housework and dating – with one finding being that quiet, soothing music and energetic music were both popular for romance, possibly indicating different preferences between genders or age groups. 

That work inspired her to look more systematically at the seemingly intangible topic of emotion in music. Dr Hu developed a model that was trained to recognise emotion in songs based on data analytics that aggregated musical aspects – such as rhythm, harmony and lyrics – with people’s responses to the music on social media. This model was then tested on new songs and had a nearly 70 per cent accuracy rate in matching song to mood. 

“Accuracy is a hard problem in this field because different people may respond differently when asked how they feel about the same piece of music. Some people even argue that each piece can express multiple emotions. So about 70 per cent accuracy is a very good performance,” she said.

Key and tempo effects 

More recently, Dr Hu has been looking at how to use the algorithms of music to enhance learning, which aligns with the mission of her Faculty and her division, the Academic Unit of Human Communication, Development, and Information Sciences. 

An early result from that research is that when people read while listening to the background music of their choice, they tend to read more slowly but to find the experience more pleasurable. Another experiment that manipulated different aspects of the music, such as tempo and major or minor mode, found fast-paced music in a minor key helped participants be more engaged in reading than a slow-paced piece in a major key. 

Dr Hu is also using wearable devices to track people’s physiological responses to music selected by researchers, particularly heart rate and blood pressure. After each song, listeners were asked to rate their emotion and answer other questions. The aim was to check the accuracy of the music information retrieval model. 

“We found that adding physiological signals could improve the accuracy of the model’s recognition of emotion in music, but not as accurately as the users’ personality assessments, which were carried out before the experiment,” she said. Users were assessed on such things as extroversion, openness and conscientiousness. 

“This has led us to hypothesise that the effects of music recommendations on things like background music reading may differ for different personalities.”

Just getting started 

She plans to assess the effects of music on learning mathematics and writing, too, and she is analysing interviews with university students about what music means to them. 

Dr Hu noted that music information retrieval is still a young field, but with great potential. Other scholars are drawing on her initial findings by using music algorithms to help Parkinson’s disease patients find their pace when walking, to understand the rhythms of the heart, and to identify patterns and differences in birdsong over time. 

Given that potential, she is keen to have more scholars join her field, particularly women. 

“This area is dominated by computer scientists and engineers and most of them are men. I served on the board of the International Society for Music Information Retrieval for six years and we are striving to achieve gender balance. I have also been a mentor for several years. We welcome newcomers – not necessarily in terms of age but in being new to the field. We encourage them to contribute,” she said.