Hearing aid technology that scans facial movements can lip read through masks

A hearing aid technology has been developed that scans your facial movements and uses an artificial intelligence (AI) to work out what is being said.

Developed by engineers at the University of Glasgow, the system is even able to read the lips of people who are wearing a mask.

The team trained algorithms with data collected by scanning people’s faces with radar and Wi-Fi signals while they were speaking.

This allowed the system to correctly interpret speech up to 95 per cent of the time for unmasked lips, and up to 83 per cent of the time with a mask.

If integrated into hearing aids, it could help deaf and hard-of-hearing people to focus on sounds more easily in noisy environments.

‘Around five percent of the world’s population – about 430 million people – have some kind of hearing impairment,’ said lead author Dr Qammer Abbasi.

‘Hearing aids have provided transformative benefits for many hearing-impaired people.

Developed by engineers at the University of Glasgow, the system is even able to read the lips of people who are wearing a mask

The team trained algorithms with data collected by scanning people’s faces with radar and Wi-Fi signals while they are speaking

The engineers first set out to train machine learning and deep learning algorithms how to recognise lip and mouth movements associated with each vowel sound. This involved asking male and female volunteers to repeat the five vowel sounds, A, E, I, O and U – while unmasked and wearing a surgical mask (stock image)

WHAT IS ‘MACHINE LEARNING’?

Machine-learning algorithms use statistics to find patterns in massive amounts of data – like numbers, words, images or clicks.

It powers many of the services we use today – including recommendation systems like those on Netflix, YouTube, and Spotify; search engines like Google; social-media feeds like Facebook and Twitter; and voice assistants like Siri and Alexa.

In all of these instances, each platform is collecting as much data about you as possible – what genres you like watching, which links you are clicking, which statuses you are reacting to – and using machine learning to make a highly-educated guess about what you might want next.

Or, in the case of a voice assistant, about which words match best with the sounds coming out of your mouth.

Source: MIT Technology Review

‘A new generation of technology which collects a wide spectrum of data to augment and enhance the amplification of sound could be another major step in improving hearing-impaired people’s quality of life.

‘With this research, we have shown that radio-frequency signals can be used to accurately read vowel sounds on people’s lips, even when their mouths are covered.’

Current hearing aids assist hearing-impaired people by amplifying all ambient sounds around them.

While this is helpful, it can be difficult for users to focus in on specific sounds in noisy situations, like while in conversation with one person at a party.

To overcome this, ‘smart’ hearing aids have been developed that collect lip-reading data using a camera, that are used alongside the conventional audio amplification.

However, collecting video footage of people without their explicit consent raises concerns for individual privacy.

The cameras are also unable to read the lips of those wearing a face covering, either for religious, cultural or health purposes.

In their study, published today in Nature Communications, the researchers outline a facial scanning technique that could work as an alternative to the camera.

They first set out to train machine-learning and deep-learning algorithms how to recognise lip and mouth movements associated with each vowel sound.

This involved asking male and female volunteers to repeat the five vowel sounds, A, E, I, O and U – while unmasked and wearing a surgical mask.

While they did this, and when their lips were still, their faces were scanned using radio-frequency signals from a radar sensor and a Wi-Fi transmitter.

This accumulated into 3,600 samples of data which were used to teach the algorithms to read masked and unmasked users’ vowel formations.

Current hearing aids assist hearing-impaired people by amplifying all ambient sounds around them. While this is helpful, it can be difficult for users to focus in on specific sounds in noisy situations, like while in conversation with one person at a party (stock image)

The Wi-Fi data was correctly interpreted by the learning algorithms up to 95 per cent of the time for unmasked lips, and 80 per cent for masked.

Meanwhile, the radar data was interpreted correctly up to 91 per cent without a mask, and 83 per cent of the time with a mask.

‘Given the ubiquity and affordability of Wi-Fi technologies, the results are highly encouraging which suggests that this technique has value both as a standalone technology and as a component in future multimodal hearing aids,’ said Dr Abbasi.

As this system preserves privacy by collecting only radio-frequency data with no video footage, it is hoped it could be installed into smart hearing aids in future.

£400 XRAI smart glasses turn audio into captions to let deaf people ‘SEE’ conversations

A new pair of smart glasses has been launched for people who are deaf or suffer from hearing loss.

The glasses, called XRAI Glass, use augmented reality to transform audio into captions that are instantly projected in front of the wearer’s eyes.

This software converts audio into a subtitled version of conversation, which then appears on the screen of the glasses.

Thanks to voice recognition capabilities, the glasses can even identify who’s speaking, and will soon be able to translate languages, voice tones, accents and pitch, according to XRAI Glass.