Project 2 - Multimodality and optimisation of cognitive resources

Image ©

Cognitive listening effort in adverse conditions is increasingly conceived as a crucial component of speech communication and language learning. Project 2 will explore various means to make communication easier and help children with HI to optimise their cognitive resources for interaction and learning. Multimodality is a primary means to provide enhanced input, associating the degraded auditory input to vision (for both lipreading and spatial hearing, and with the possible addition of manual linguistic cues) but also somatosensory information on the speaker’s gestures. Various tools for cognitive training together with innovative speech technologies will also be explored to enhance communication and learning.

The project Multimodality and optimisation of cognitive resources involves six tasks exploring various facets of multimodal speech communication, cognitive training, and evaluation of software and human-machine technologies.

This project will be lead by Jean-Luc Schwartz at the CNRS-GIPSA lab, Université Grenoble-Alpes. The Comm4CHILD beneficiaries Centre Comprendre et Parler, CNRS-GIPSA, KU Leuven, University of Oslo, OTIC, and Université libre de Bruxelles as well as the partner organisations HUDERF and IVèS will contribute to project 2. Topic 2.1 and 2.2 will study manual gestures in CS in speech comprehension and in speech production. In Topic 2.3-2.5, the researchers will explore various tools for cognitive evaluation and cognitive enhancement, aiming at enhancing the abilities of children with hearing impairment – and also deaf-blind children – to monitor listening effort in adverse conditions, facilitate communication and improve learning. The researcher working on Topic 2.6 will develop augmented communication systems and algorithmic tools related to sign language or audio-visual speech.

Project leader