Multimodality and optimisation of cognitive resources

Cognitive listening effort in adverse conditions is increasingly conceived as a crucial component of speech communication and language learning. This work package will explore various means to make communication easier and help children with hearing impairment to optimise their cognitive resources for interaction and learning. Multimodality is a primary means to provide enhanced input, associating the degraded auditory input to vision (for both lipreading and spatial hearing, and with the possible addition of manual linguistic cues) but also somatosensory information on the speaker’s gestures. Various tools for cognitive training together with innovative speech technologies will also be explored to enhance communication and learning.

5. Temporal course of auditory, labial, and manual signals in Cued Speech perception

6. Exploiting the somatosensory function in hearing impaired subjects for understanding and rehabilitating the perceptuo-motor loop in speech communication

7. Improving integration of audio-visual speech cues in children with hearing impairment

8. Effortful listening, cognitive energy, and learning in children fitted with cochlear implant

9. Development of working memory capacity, phonological and lexical skills, and executive functions in children with hearing impairment and deafblind children

10. Automatic recognition and generation of Cued Speech using deep learning