According to the German Association of the Deaf (DSB), some 15 million Germans are hard of hearing. Hearing impaired people can find it difficult to follow a discussion in a loud environment, particularly when the conversation involves several people. Often they have trouble picking out individual voices. This is because they are unable to make use of what is known as the cocktail party effect: the ability to focus on one speaker and filter out any other background noises.

Current hearing aids are unable to establish the special link between the ear and the brain that accounts for the process of selective hearing. “For people with normal hearing, the connection between the ear and the brain is unimpaired,” explains Dr. Axel Winneke, research fellow at the Fraunhofer Institute for Digital Media Technology IDMT in Oldenburg. “This is how they know which direction to face whenever they hear a signal. For hearing-aid wearers, this ability is severely restricted. Even high-end devices are unable to pinpoint the source of an acoustic signal of interest. We therefore need to retrieve this information from the brain. Using an electroencephalograph (EEG), we can analyze the brain activity of a hearing impaired person and determine who it is they are listening to.” This EEG analysis is conducted by Fraunhofer IDMT-HSA and the University of Oldenburg. Together with partners from industry and research (see “The mEEGaHStim project”), Winneke and colleagues are developing a system designed to improve the intelligibility of speech in busy environments for the hard of hearing. This technology is based on a combination of EEG, audio-signal processing, and electrostimulation of the auditory cortex. It functions as follows: an EEG measures brain activity in order to determine the direction in which the hearing impaired person is trying to hear – i.e., toward the source of the signal – thereby forming an interface between the brain and a microprocessor. This information is transmitted to the hearing aid, which focuses a directional microphone – known as a beamformer – in the corresponding direction. This beamformer amplifies the specific audio signal that the hearer is trying to focus on, simultaneously filtering out all other sources of noise, including other voices. The third element of the system is a process from the field of neuroscience called transcranial electrostimulation (tES). This employs a very low electrical current to stimulate the auditory cortex on the basis of this voice signal, thereby improving the intelligibility of the person talking. The stimulation process and requisite hardware are being developed by the company neuroConn GmbH in partnership with the University of Oldenburg.

Hearables of the future

Design studies have already been carried out in order to visualize what these new hearables might look like. The concept and the design are intended to emphasize – rather than conceal – the brain’s interaction with the device. At the same time, the design seeks to highlight the benefits for the wearer, thereby consciously steering against the still widespread stigma attached to the wearing of a hearing aid. In the future, components developed for the project, including sensor technology, could be incorporated in a wearable headpiece. Another option would be to retrofit existing hearing aids with these new modules and an EEG sensor. “The current prototype is not yet available in the form of a portable hearing aid,” Winneke explains. “For that, it still needs to be substantially miniaturized.” Early trials with the device, which were conducted with people without a hearing impairment, have shown that the principle works well. Studies with people hard of hearing are now being planned.

Mobile neurotechnologies

A wearable EEG headpiece could also be used for other applications, such as measuring the listening effort required by employees in the workplace. The technology could also be used in a medical context – in neurology, for example, to monitor conditions such as epilepsy. “And using a wearable EEG device would make it possible to observe patients outside of the clinical environment,” says Winneke. “In the mEEGaHStim project, we use the technology to measure brain activity in order to control a hearing aid, but it could also be used to analyze brain activity in people with neurological disorders.” Winneke belongs to the Mobile Neurotechnologies group at Fraunhofer IDMT. This group is working on EEG-based multisensor platforms for use in concrete scenarios where the analysis of brain activity can provide support – for example, in medical applications or safety-critical work situations.

Hearing, speech and audio technology HSA at the Fraunhofer Institute for Digital Media Technology IDMT in Oldenburg

The objective of the Hearing, Speech and Audio Technology Division (HSA) of the Fraunhofer Institute for Digital Media Technology IDMT is to transpose scientific findings related to hearing perception and human-technology interaction into technological applications. Its applied research priorities are the enhancement of sound quality and speech intelligibility, personalized audio reproduction and acoustic speech recognition and event detection with the help of artificial intelligence. A further focus is the use of mobile neurotechnologies, which facilitate the recording of brain activity and utilization of the resulting data outside the laboratory too. Application fields include consumer electronics, transport, the automotive sector, industrial production, security, telecommunications and healthcare. Through scientific partnerships, Fraunhofer IDMT-HSA has close links with the Carl von Ossietzky University of Oldenburg, Jade University and other institutions engaged in hearing research in Oldenburg. Fraunhofer IDMT-HSA is a partner in the »Hearing4all« Cluster of Excellence.

The Division Hearing, Speech and Audio Technology HSA is funded in the program »Vorab« by the Lower Saxony Ministry of Science and Culture (MWK) and the Volkswagen Foundation for its further development.