Communication in the maritime environment is important for safety but does not always run smoothly. This can have catastrophic consequences. After all, 90 percent of all accidents at sea are caused by human error, and half of these by problems with communication. In collaboration with Jade University of Applied Sciences (Elsfleth Campus) and partners from industry, Fraunhofer IDMT in Oldenburg, Germany, has been working for some time on improving safety-relevant voice communication, particularly at sea.

»In addition to developing chatbots for training purposes and recruitment procedures, the focus is on systems for monitoring voice commands on the ship’s bridge. For this purpose, we have trained our speech recogniser’s AI with special vocabulary, known as »Nautical Terms and Phrases« or »maritime English«, says Jan Wellmann, Head of Automatic Speech Recognition at the Oldenburg Branch for Hearing, Speech and Audio Technology HSA of Fraunhofer IDMT.

ELNAV, a start-up company based in the Croatian port city of Split, is developing what it calls a »Helm Order Monitor«. This electronic device uses speech recognition technology from Fraunhofer IDMT in Oldenburg in conjunction with data received from the ship’s sensors and monitors whether the commands issued are clear, confirmed and correctly executed.

Hrvoje Mihovilović, ELNAV’s founder and CEO says: »In the beginning, the biggest obstacle for ELNAV was deciding whether to develop our own speech recognition system or find a partner. We needed a robust speech detector that would work reliably even under difficult acoustic conditions. We soon discovered that Fraunhofer IDMT was developing a speech recognition model for maritime communications and therefore an ideal partner for the development of our Helm Order Monitor.«

Technological challenges

The partners faced three main technological challenges. The first was the signal degradation caused when using far-field microphones in speech-processing applications. The noise level on the bridge should not interfere with verbal communication, mask audible alarms or be uncomfortable for bridge personnel: the ambient noise level on the bridge in calm weather should not exceed 65 dB(A). To solve this problem, microphone array signal processing is presented as an alternative.

The accuracy of Automatic Speech Recognition (ASR) under different noise conditions and at various distances was a further challenge. To make the ASR system more robust in noisy environments, deep neural networks (DNN) are used to enhance speech.

A third challenge was catering for the endless variations of the English language – from different regional accents to idiosyncratic use of grammar and vocabulary. Here, machine learning is used to create a single, comprehensive language pack which is accurate and encompasses as many variations of English as possible.

»The latest report on European Maritime Safety (EMSAFE) sees digitalization and increasing automation as a major opportunity. However, the technologies could also bring new challenges for safety as well as crew training. The changes required for this are supported by our speech recognition system. It can be used to learn, test and monitor maritime English in practice«, emphasizes Dr. Jens Appell, head of the Oldenburg branch of Fraunhofer IDMT.

Hearing, Speech and Audio Technology HSA at Fraunhofer IDMT in Oldenburg

Founded in 2008 by Prof. Dr. Dr. Birger Kollmeier and Dr. Jens-E. Appell, the Fraunhofer Institute for Digital Media Technology IDMT’s Branch for Hearing, Speech and Audio Technology HSA stands for market-oriented research and development with a focus on the following areas:

• Speech and event recognition
• Sound quality and speech intelligibility
• Mobile neurotechnology and systems for networked healthcare

With in-house expertise in the development of hardware and software systems for audio system technology and signal enhancement, over 100 employees at the Oldenburg site are responsible for transferring scientific findings into practical, customer-oriented solutions.

Through scientific cooperation, the institute is closely linked to the Carl von Ossietzky University, Jade University of Applied Sciences, and the University of Applied Sciences Emden/Leer. Fraunhofer IDMT is a partner in the »Hearing4all« cluster of excellence.

The Oldenburg Branch for Hearing, Speech and Audio Technology HSA is funded in the program »Vorab« by the Lower Saxony Ministry of Science and Culture (MWK) and the Volkswagen Foundation for its further development.