NUS Tech Feature – Dr. Wang Ye

What does music have to do with wellness and health?

In one aspect of music, singing is known to release the ‘happy’ chemical, endorphins, to make one feel uplifted and good. In our latest instalment of the BLOCK71 Tech Talk, we learn from Associate Professor Dr Wang Ye of the National University of Singapore (NUS) School of Computing, who uses his research in music and mobile computing for health and learning to help those with disabilities. His work focuses on finding the intersection between these four elements, and his technology can be applied to speaking and walking. Dr Wang believes that the lack of such capabilities can be a frustrating problem for those with disabilities, and is keen on helping them using his expertise

Dr. Wang Ye, an Associate Professor at the NUS School of Computing

Who are you and what do you do?

I am Wang Ye, an Associate Professor at the NUS School of Computing Science. The research focus in my lab is basically in a section between computing, music, health care and learning. In one sentence, we are doing research on music and Computing for health and learning. Many IT theories and systems have attempted to model or mimic the human capacity in areas such as speech recognition and musical information retrieval. We try to use signal processing to model the auditory system. Then we use machine learning to mimic what happens to the brain when we perceive music. There are two important applications that I care about, speaking and walking, which are fundamental capabilities of a human being.

For people without any disability, speaking and walking are capabilities that we take for granted but for those suffering from health problems like Aphasia or stroke, they immediately lose the ability to speak. Walking is also a very important proficiency for a person to move from A to B. If you lose the ability to walk and have to rely on a wheelchair, it can become a very big frustration.

What are some noteworthy or interesting projects that you have worked on?

Language and Social Interaction

When I had my sabbatical at Harvard Medical School in Boston, they had developed a form of web-based therapy. Basically, they recorded a set of practice videos that the patients can follow, in which patients listen to some phrases and repeat them. However, the problem with this system is the lack of feedback. It is hard to ascertain whether the patients are learning the phrases correctly.

I then proposed a project called SLIONS, which makes use of singing and listening to improve natural speaking. It is an iPad application that can provide different kinds of visualization and feedback for the patients when they speak the words correctly. The app is available on the Apple App Store and Google Play Store.

Some of my other projects include Melodic Intonation Therapy, where singing is used to help stroke patients speak again. Another project is SECCIMA, which stands for Singing and Ear training for Children with Cochlear Implants with a Mobile Application. SECCIMA training takes the form of a colourful xylophone, and users have to relate musical notes to different colours and positions. As the platform is easily accessible via a mobile device, children with cochlear implants can undergo auditory training more conveniently as well as in an interactive manner.

Watch the introductory Video to SECCIMA here.

Movement and Wellness

For movement and wellness, we particularly focused on Parkinson’s Disease. Some common symptoms for Parkinson’s are tremor, rigidity and short shuffling steps. Hence, we try to develop affordable music and wearable technology for health and rehabilitation. Gait refers to the way a person walks and we do a gait analysis to understand how we can improve the patients’ walking ability. We tackle problems like Auditory Tempo Stability and Synchronisation between gait and musical beats. Imagine stepping on a musical beat when you are walking, it is simply much easier. Even for soldiers, they march according to a musical beat. Similarly, we can use the same principle to train patients to walk better. With more advanced AI technology today, we want to try combining human intelligence and artificial intelligence to come up with affordable solutions for these patients.

To improve our data collection in Gait analysis, we developed a device called Mobility Analytics (MANA) sensors. The sensor is designed like a smartwatch and can be worn on the wrist, or the sensor can be placed in the shoes. This design will allow us to have a better quantification of the gait compared to our previous smartphone app. We realised that using a smartphone device alone is not good enough for accurate data collection because the smartphones were placed in awkward positions, for example when it is strapped on the patient’s waist.

Mobility Analytics (MANA) Sensors

Another equipment that we needed was a portable mat device for Gait Analysis. However, GAITRite, the golden standard for this device costs $70,000. It is very expensive and that’s why we designed and built one by ourselves that has the same function but costs less than $10. We tested it and the accuracy was pretty good. We carried out the MANA data collection in one of the best hospitals in Shanghai with patients suffering from Parkinson’s disease.

MANA in action: patients walking on the mat designed for Gait analysis.

Music creation for Health and Entertainment

Another project that we are doing right now is the Motion-Initiated Music Ensemble with Sensors (MIMES) project where we use a sensor designed like a smartwatch, to detect your movement and control the sound produced. With this, we can create music and songs with an ensemble of people wearing these sensors.

Watch how music and different instrumental sounds can be created through the use of these motion sensors.

What an exciting way to link music, healthcare, and technology together! Indeed, this seems like a promising field in which science and the arts can meet and create a social impact. If you have any questions about how music and technology can work together in healthcare, feel free to contact Dr. Wang Ye at .

Watch the video interview with Dr. Wang Ye here:

For startup updates straight to your inbox, subscribe to our weekly newsletter here.