Topic 3.2 - Multimodal communication (ESR12)

Image ©

Topic 3.1, Multilingual language development, is hosted at the University of Leeds and supervised by Ruth Swanwick and Edward Killan.

This task examines how multimodal aspects of communication can facilitate interaction and support language learning for children with hearing impairment (HI), in situations where there are sensory and communication ‘asymmetries’ in terms of language access, experience and skills and the use of hearing technologies. We hypothesise that individuals with different visual/gestural and auditory/oral experience of language draw on a complex repertoire of multimodal (or non-verbal) communicative strategies to co-construct situated understanding beyond cultural and linguistic barriers, to fulfil their communicative needs in specific situations of daily life, and that the use of hearing technologies influences the development of these resources over time (Kusters et al 2017). We will examine the daily life interactions between children with and without HI and adults in a) family and social milieu and b) the education context. We will undertake close scrutiny of video-recorded data of situated sign/spoken language interactions among children with and without HI and adults who do not share the same linguistic resources and we will undertake multimodal analysis of this interactive data. For this purpose the early stage researcher (ESR) will be trained to use the software tool ELAN (EUDICO Linguistic Annotator) that is a specifically designed for the analysis of language, SL, and gesture, for purposes of annotation, analysis and documentation.

In topic 3.2, we aim to will reveal the potential of multimodal strategies to facilitate interaction between children with HI and their family members, teachers and peers and for the development of their language learning; how multimodal communication strategies are influenced by context, the interactional situation, audiological variables and technologies and how they change over time. We envisage applications for early intervention with families, education, health, and interpreting and communication practices in deaf/hearing communities more generally.

Part of the research on topic 3.2 will take place at the Bradford Teaching Hospitals, Yorkshire Auditory Implant Centre, and at the Université libre de Bruxelles. The first secondment will facilitate the collection and analysis of interactional data among children and families whereas the second aims at sharing analytical processes, including the use of ELAN for annotation, analysis, and documentation.

Project leader