Co-speech Gestures and Prosody as Multimodal Markers of Information Structure

Project Participants

Frank Kügler

X

Prof. Dr. Frank Kügler
(Principal Investigator)

Goethe University Frankfurt
kuegler@em.uni-frankfurt.de

Frank Kügler is Professor of Linguistics at Goethe University Frankfurt. His research interests are in cross-linguistic prosody from various different perspectives such as the expression and modelling of information structure, prominence and sentence mode in typologically unrelated languages, the interaction of tone and intonation, and prosodic phrasing and recursive prosodic constituents at the prosody-syntax interface. His recent interests include also the gestural marking of prosody. He has worked on the prosody of a number of typologically diverse languages including Mandarin, Hindi, Akan, Tswana, Ghanaian English, Yucatec Maya, Hungarian, and German among others.

Selected publications

  • Grice, Martine & Kügler, Frank (2021) Prosodic Prominence – A Cross-Linguistic Perspective. Language and Speech, 64, 2, 253–260. https://doi.org/10.1177/00238309211015768
  • Kügler, Frank (2016) Tone and intonation in Akan. In: Downing, Laura J. & Rialland, Annie (eds) Intonation in African Tone Languages. Berlin: Mouton de Gruyter, 89-129.
  • Kügler, Frank, Baumann, Stefan & Röhr, Christine T. (2022) Deutsche Intonation, Modellierung und Annotation (DIMA) – Richtlinien zur prosodischen Annotation des Deutschen. In Schwarze, Cordula & Grawunder, Sven (eds.) Transkription und Annotation gesprochener Sprache und multimodaler Interaktion: Konzepte, Probleme, Lösungen. Tübingen: Narr, 23-54.
  • Kügler, Frank & Sasha Calhoun. 2020. Prosodic Encoding of Information Structure: A typological perspective. In Carlos Gussenhoven & Aoju Chen (eds.), The Oxford Handbook of Language Prosody, 453–467. Oxford: OUP.
  • Kügler, Frank & Caroline Féry. 2017. Postfocal downstep in German. Language and Speech 60(2). 260–288.

Pilar Prieto

X

Prof. Dr. Pilar Prieto
(Principal Investigator)

Universitat Pompeu Fabra (UPF), Barcelona
pilar.prieto@upf.edu

Pilar Prieto is an ICREA Research Professor at the Department of Translation and Language Sciences at Universitat Pompeu Fabra, Barcelona, Catalunya. Her research focuses on the communicative role of prosody and gesture in language, as well as their significance in language development and second language learning. In the last years, she has worked on the multimodal marking of Information Structure, in both adult and children’s discourse. She currently serves as associate editor of the journals Language and Speech and Frontiers in Communication, and is coediting a special issue of the “Language and Cognition” journal on Multimodal Prosody.

Selected publications

  • Brown, Lucien & Pilar Prieto. 2021. Gesture and prosody in multimodal communication. In Michael Haugh, Dániel Z. Kádár & Marina Terkourafi (eds.), The Cambridge Handbook of Sociopragmatics, 430–453. Cambridge: CUP.
  • Esteve-Gibert, Núria & Pilar Prieto. 2013. Prosodic structure shapes the temporal realization of intonation and manual gesture movements. Journal of Speech, Language, and Hearing Research 56(3). 850-864.
  • Prieto, Pilar, C. Puglesi, J. Borràs-Comes, E. Arroyo & J. Blat. 2015. Exploring the contribution of prosody and gesture to the perception of focus using an animated agent. Journal of Phonetics 49(1). 41-54.
  • Rohrer, P., Florit-Pons, M., Vilà-Giménez, I., Prieto, P. (2022). “Children use non-referential gestures in narrative speech to mark discourse elements which update common ground.” Frontiers in Psychology 12:661339. doi: 10.3389/fpsyg.2021.661339
  • Vanrell, M.M., Stella, A., Gili-Fivela, B., & Prieto, P. (2013). Prosodic manifestations of the Effort Code in Catalan, Italian, and Spanish contrastive focus. Journal of the International Phonetics Association 43(2), pp.195-220.

Alina Gregori

X

Alina Gregori
(PhD Candidate)

Goethe University Frankfurt
gregori@lingua.uni-frankfurt.de

Alina Gregori received her Bachelors (2021) and Masters (2022) degree in theoretical linguistics at Goethe University Frankfurt with a focus on Phonology. She started working as a PhD student in the project MultIS in October 2022. Within the project, she investigates the prosody-gesture link in communication and the impact of information structure (focus, topic, givenness) on the synchronization of gestures and prosodic entities in German. The bigger picture of the project includes a comparative analysis of German and Catalan with regard to the multimodal marking of information structure. A central aim in MultIS is the empirical approach to previously established prosody-gesture models, considering experimental as well as spontaneous speech utterances. Alina Gregoris MA thesis (title: “Co-speech Gestures, Information Structure and Prosody: A Corpus Study on Prominence Peak Alignment”) served as a pilot and preparation for MultIS.

Selected publications

  • Henlein, A., A. Bauer, R. Bhattacharjee, A. Ćwiek, A. Gregori, F.Kügler, J. Lemanski, A. Lücking, A. Mehler, P. Prieto, P. G. Sánchez-Ramón, J. Schepens, M. Schulte-Rüther, S. R.Schweinberger & C. I. von Eiff. 2024. An Outlook for AI Innovation in MultimodalCommunication Research. Lecture Notes in Computer Science, Springer Cham.
  • Gregori, A., P. G. Sánchez-Ramón, P. Prieto & F. Kügler. 2024. Prosodic and gestural marking of focus types in Catalan and German. In: Proc.Speech Prosody 2024, 891-895, doi: 10.21437/SpeechProsody.2024-180 
  • Kügler, F. & A. Gregori. (2023). Iconic Gestures in Focus – Synchronizationof Prosody and Gestures in Prominence. In: Skarnitzl, R. & Volín, J. (eds)Proceedings of 20th ICPhS 2023, Prague, Czech Republic August 2023. 4125-4129(ID 232). Guarant International.
  • Gregori, A., F. Amici, I. Brilmayer, A. Ćwiek, L. Fritzsche, S. Fuchs, A. Henlein, O. Herbort, F. Kügler, J. Lemanski, K. Liebal, A.Lücking, A. Mehler, K.Tien Nguyen, W. Pouw, P. Prieto, P. L. Rohrer, P. G. Sánchez-Ramón, M. Schulte-Rüther, P. Schumacher, S. Schweinberger, V. Struckmeier, P. C. Trettenbrein & C. I. von Eiff. (2023). A Roadmap for Technological Innovation in Multimodal Communication Research. In: Duffy, V.G. (eds) Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. HCII 2023. Lecture Notes in Computer Science, vol 14029. 402-438. Springer, Cham. https://doi.org/10.1007/978-3-031-35748-0_30  .
  • Gregori, Alina & Kügler, Frank (2022). An Empirical Investigation on the Perceptual Similarity of Prosodic Language Types. In Proc. 1st International Conference on Tone and Intonation (TAI), 209-213, doi: 10.21437/TAI.2021-43 .

Paula G. Sánchez Ramón

X

Paula G. Sánchez Ramón
(PhD Candidate)

Universitat Pompeu Fabra (UPF), Barcelona / Goethe University Frankfurt
paulaginesa.sanchez@upf.edu

Paula G. Sánchez-Ramón is a PhD student developing her research under an international joint cotutelle between Universitat Pompeu Fabra (Barcelona, Catalonia) and Goethe Universität (Frankfurt am Main, Germany). She holds a B.A. in Primary Education and Teaching (Universitat Jaume I) with a specialization in Teaching English as a Foreign Language (Universidad Internacional de La Rioja), and an M.A. in Learning Difficulties and Language Disorders with a concentration in Speech and Language (Universitat Oberta de Catalunya). As a member of the project MultIS, her research focuses on the prosodic and gestural marking of information structure in Catalan-speaking adults.

Project Description

Information structure (IS) can be conceived of as a ‘cognitive domain’ interacting with the linguistic modules syntax, phonology, and morphology to control, update, and infer interlocutors’ common beliefs (Zimmermann & Féry 2010). While the interaction between IS and syntactic and phonological/prosodic patterns has been widely investigated, less is known about the visual marking of IS. From a prosodic standpoint, it is well-known that languages use prosodic features like pitch accentuation to convey IS (Calhoun & Kügler 2020 for a review). For example, in Germanic languages, given referents tend to be deaccented more often than new referents (Baumann & Grice 2006; Baumann & Schumacher 2020; Féry & Kügler 2008; Ladd 2008). By contrast, Romance languages have been reported to place focused constituents in phrase-final position (Zubizarreta 1998) and to lack systematic deaccentuation of given referential expressions (Ladd 2008; Swerts et al. 2002). From a multimodal perspective, it is a well-established fact that co-speech gestures typically align with pitch accented syllables across languages (McNeill 1992; Kendon 2004; Loehr 2004). Even though some gesture studies have hinted at the relationship between the presence of “beat gestures” and the marking of focused information, to our knowledge very few empirical investigations have assessed whether IS has an impact on (a) the presence of co-speech gestures; and (b) the gesture-speech timing patterns, as well as the prominence levels of the whole gesture.

The main goal of this project is to assess the multimodal marking of IS from a crosslinguistic point of view. The two languages under investigation, Catalan and German, share the use of pitch accents for prosodic prominence marking of IS. They differ however in their positioning of accent patterns in relation to IS (variable focus position in German vs. tendency for rightmost focus position in the utterance in Catalan). Starting from the well-established coordination between co-speech gestures and prosodic prominence, the general question we ask is whether different information structural categories are signalled not only by prosodic events but also through the use of co-speech gestures. The project will use both controlled experimental materials as well as speech corpora of spontaneous discourse to analyse the interface between IS marking and prosodic and gestural features. We expect that in both Catalan and German (a) co-speech gestures will be associated more frequently with pitch accents that mark focus domains and novel information; (b) that the time alignment of cospeech gestures with speech will be directly affected by IS domains; and (c) that gesture amplitude patterns are directly correlated with degrees of prosodic prominence. In general, we expect that the presence of co-speech gesture will not be exclusively predicted by the presence/absence of pitch accentuation, but rather will have a unique role that will be predicted by IS

Project Activities

Publications

Henlein, A., A. Bauer, R. Bhattacharjee, A. Ćwiek, A. Gregori, F.Kügler, J. Lemanski, A. Lücking, A. Mehler, P. Prieto, P. G. Sánchez-Ramón, J. Schepens, M. Schulte-Rüther, S. R.Schweinberger & C. I. von Eiff. 2024. An Outlook for AI Innovation in MultimodalCommunication Research. Lecture Notes in Computer Science, Springer Cham.

Gregori, A., P. G. Sánchez-Ramón, P. Prieto & F. Kügler. 2024. Prosodic and gestural marking of focus types in Catalan and German. In: Proc.Speech Prosody 2024, 891-895, doi: 10.21437/SpeechProsody.2024-180

Kügler, F. & A. Gregori. 2023. Iconic Gestures in Focus – Synchronizationof Prosody and Gestures in Prominence. In: Skarnitzl, R. & Volín, J. (eds)Proceedings of 20th ICPhS 2023, Prague, Czech Republic August 2023. 4125-4129(ID 232). Guarant International.

Gregori, A., F. Amici, I. Brilmayer, A. Ćwiek, L. Fritzsche, S. Fuchs, A. Henlein, O. Herbort, F. Kügler, J. Lemanski, K. Liebal, A.Lücking, A. Mehler, K.Tien Nguyen, W. Pouw, P. Prieto, P. L. Rohrer, P. G. Sánchez-Ramón, M. Schulte-Rüther, P. Schumacher, S. Schweinberger, V. Struckmeier, P. C. Trettenbrein & C. I. von Eiff. 2023. A Roadmap for Technological Innovation in Multimodal Communication Research. In: Duffy, V.G. (eds) Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. HCII 2023. Lecture Notes in Computer Science, vol 14029. 402-438. Springer, Cham. https://doi.org/10.1007/978-3-031-35748-0_30.

The MULTIFOCUS CORPUS: An audiovisual database for the elicitation of focus types and semi-spontaneous speech and gestures in Catalan and German

Conference contributions

Kügler, F., Prieto, P., Gregori, A. & Sánchez-Ramón, P. (2022). Co-speech gestures and prosody as multimodal markers of information structure [Oral Presentation]. ViCom Annual Meeting. Bad Homburg, Germany. (November)

Kügler, F. & Gregori, A. (2023, March 28). Co-speech gestures and prosody as multimodal markers of information structure – MultIS. [Oral Presentation]. ViCom Theory Workshop. Königstein, Germany.

Sánchez-Ramón, P., Gregori, A., Prieto, P. & Kügler, F. (2023, April 26-28). The impact of focus types on the prosody-gesture link in two Romance and Germanic languages: a focus elicitation production study [Poster Presentation]. The 1st International MultiModal Communication Symposium (MMSYM), Universitat Pompeu Fabra, Barcelona, Spain.

Sánchez-Ramón, P., Gregori, A., Prieto, P. & Kügler, F. (2023, June 2-4). Focus types and the prosody-gesture link in Catalan and German: A production study [Poster Presentation]. Phonetics and Phonology in Europe (PaPE) 2023, Nijmegen, The Netherlands.

Gregori, A., Sánchez-Ramón, P., Prieto, P. & Kügler, F. (2023, July 9-14). Focus types and the prosody-gesture link in Catalan and German: A production Study [Oral Presentation]. The 18th International Pragmatics Conference (IPrA), Brussels, Belgium.

Sánchez-Ramón, P., Gregori, A., Kügler, F. & Prieto, P. (2023, July 21). El marcatge prosòdic i gestual dels tipus de focus en llengua catalana: Un estudi de producción [Oral Presentation]. Workshop sobre la prosòdia del català (14a edició), Universitat Pompeu Fabra, Barcelona, Spain.

Kügler, F. & Gregori, A. (2023, August 10). Iconic Gestures in Focus – Synchronization of Prosody and Gesture in Prominence [Poster Presentation]. 20th International Congress of Phonetic Sciences (ICPhS). Prague, Czech Republic.

Gregori, A., Sánchez-Ramón, P., Kügler, F. & Prieto, P. (2023, October 12-13). Multimodal marking of prominence across focus types in German and Catalan [Oral Presentation]. Voices in Context Workshop, Cologne, Germany.

Kügler, F., Prieto, P., Gregori, A. & Sánchez-Ramón, P. (2023, November 09). Co-speech gestures and prosody as multimodal markers of information structure [Oral Presentation]. ViCom Annual Meeting. Bad Homburg, Germany.

Sánchez-Ramón, P., Gregori, A., Kügler, F. & Prieto, P. (2024, January 25). On eliciting multimodal data – the challenge between controlling the context and eliciting spontaneous speech and gesture [Oral Presentation]. Roundtable on Multimodal Speech Data. Frankfurt, Germany.

Ćwiek, A., Gregori, A., Sánchez-Ramón, P., Kügler, F. & Prieto, P. (2024, February 29). Bridging the Gap – Exploring a Middle-Way Approach for Prosodic Annotation [Oral Presentation]. Workshop “Prosody in Focus – Prominence Marking from Multiple Perspectives” at the DGfS 2024. Bochum, Germany.

Gregori, A., Sánchez-Ramón, P., Prieto, P. & Kügler, F. (2024, July 2-5). Prosodic and gestural marking of focus types in Catalan and German [Poster Presentation]. Speech Prosody 2024, Leiden, The Netherlands.

Sánchez-Ramón, P., Gregori, A., Kügler, F. & Prieto, P. (2024, September 6th). La prominència prosòdica va lligada a la prominència gestual? El marcatge multimodal del focus en català [Oral Presentation]. Workshop sobre la prosòdia del català (15a edició), Universitat de les Illes Balears, Palma de Mallorca, Spain.

Sánchez-Ramón, P., Kügler, F. & Prieto, P. (2024, September 25-27). The influence of gesture presence on the prosodic realization of focus types in the Catalan language [Oral Presentation]. MMSYM 2024, Frankfurt, Germany.

Gregori, A. & Fuchs, S. (2024, September 26). Moving Meetings by Moving Prosody and Gesture. [Poster Presentation]. Second International Multimodal Communication Symposium (MMSYM 2024). Frankfurt, Germany.

Invited talks and guest lectures

Invited Lecture at the Studentische Tagung der Sprachwissenschaft (STuTS): FrankKügler in Frankfurt, Germany, 2023, May 27.

Summer school courses

Summer School Lecture: Frank Kügler at the DGfS Summerschool “Form-Meaning Mismatches in Spoken and Visual Communication” in Göttingen, Germany; 2024, August 19 – 23.

Summer School Lecture: Frank Kügler at the DGfS Summerschool “The Interaction of Gesture and Prosody” in Göttingen, Germany; 2024, August 19 – 23.

Events
DateEventOrganization
16.01. – 03.02. 2023M3D annotation WorkshopPatrick Rohrer & Pilar Prieto
26. – 28.04.2023MMSYM 2023Pilar Prieto & Paula G. Sánchez-Ramón
09. – 10.07.2023IPrA Panel “Multimodal and prosodic markers of information structure and discourse structure”Pilar Prieto & Frank Kügler
21.07.2023Catalan Prosody WorkshopPilar Prieto & Paula G. Sánchez-Ramón
25.01.2024Roundtable on Multimodal Speech DataMultIS
25. – 27.09.2024MMSYM 2024Frank Kügler & Alina Gregori
10. – 11.10.2024Workshop “Consolidating annotation standards for non-manual markers across sign language and gesture research”Pilar Prieto, Paula G. Sánchez-Ramón and other ViCom projects
May / June 2025


Workshop “Multimodal Effort”MultIS and other ViCom projects
25.06.2025PaPE Workshop: Multimodal marking of information and discourse structureFrank Kügler & Pilar Prieto
Short-term Collaborations

Testing the correlation between top-down prosodic annotation systems andbottom-up automatic annotation (2023, 2024) – Aleksandra Ćwiek (FLESH), PilarPrieto (MultIS), Frank Kügler (MultIS), Patrick Rohrer (external collaborator)

Exploring multimodal prominence in Slicing gesture sequences (2024) – SilvaLadewig (StabiGes), Frank Kügler (MultIS), Alina Gregori (MultIS), Pilar Prieto(MultIS), Paula G. Sánchez-Ramón (MultIS)

Moving Meetings by Moving Prosody and Gesture (2024) – Alina Gregori (MultIS),Susanne Fuchs (FLESH)

Theses (in progress):

Sánchez-Ramón, P. G. (in prep.) Co-speech gestures and prosody as multimodalmarkers of focus types in the Catalan language. Universitat Pompeu Fabra and Goethe Universität Frankfurt.

Gregori, A. (in prep.) Co-speech gestures and prosody as multimodal markers ofinformation structure in German. Goethe Universität Frankfurt.

Natascha Schuldes (2024). Untersuchung der Alignierung von Gesten und Prosodie und deren Einfluss auf die Natürlichkeit von Sätzen eine empirische bewertungsbasierte Erhebung. (Investigation of the alignment of gestures and prosody and their influence on the naturalness of sentences – an empirical assessment-based survey). BA thesis at Goethe University Frankfurt, supervised by Frank Kügler & Alina Gregori

Lars Defty (2024). Audiovisual prosody in child-directed and adult-directed narrative speech: speakers’ body movements and prosodic variance. MA thesis at Universitat Pompeu Fabra, supervised by Pilar Prieto & Ingrid Vilà-Giménez (Universitat de Girona)

Carla Rufí (2024). Is there a positive relationship between acoustic prominence and gesture prominence? A corpus-based study on academic speech. MA thesis at Universitat Pompeu Fabra, supervised by Pilar Prieto & Gilbert Ambrazaitis (Linneaus University)

Ulya Tütüncübasi (2023). M3D Gesture Labeling: Developing a training program for gesture annotation. MA thesis at Universitat Pompeu Fabra, supervised by Pilar Prieto & Patrick Rohrer (Radboud University)

Bethany Annal (2023). Gesture Prominence and Information Structure. MA thesis at Universitat Pompeu Fabra, supervised by Pilar Prieto & Patrick Rohrer (Radboud University)

Other

Course attendance: Alina Gregori and Paula G. Sánchez-Ramón at the M3DAnnotation Course, Barcelona, Spain; 2023, January 16th-February 2nd

Summer School Attendance: Paula G. Sánchez-Ramón at the Summerschool“‘Practical Approaches to Human Multimodal Behavior” in Nijmegen, TheNetherlands; 2023, September 4-8.

Summer School Attendance: Alina Gregori at the DGfS Summerschool“Form-Meaning Mismatches in Spoken and Visual Communication” in Göttingen, Germany; 2024, August 12 – 23.