Project Participants
Dr. Susanne Fuchs
(Principal Investigator)
Leibniz-Zentrum Allgemeine Sprachwissenschaft Berlin
fuchs@leibniz-zas.de
Susanne Fuchs (ZAS Berlin) investigates the biopsychosocial foundations of human interaction and focuses specifically on physiological processes, such as breathing and motor control. Her main areas of interests are:
1) The interplay between motion, breathing and cognition,
2) Speech preparation and pauses,
3) Multimodality and iconicity,
4) Biological and social aspects shaping individual behaviour in speech production and perception.
She uses manifold techniques, among them optitrack, inductance plethysmography, electropalatography and intraoral pressure sensors.
Selected publications
- Pouw, W., & Fuchs, S. (2022). Origins of vocal-entangled gesture. Neuroscience & Biobehavioral Reviews, 104836.
- Fuchs, S., & Rochet-Capellan, A. (2021). The respiratory foundations of spoken language. Annual Review of Linguistics, 7(1), 13-30.
- Ćwiek, A., Fuchs, S., Draxler, C., Asu, E. L., Dediu, D., Hiovain, K., … & Perlman, M. (2021). Novel vocalizations are understood across cultures. Scientific Reports, 11(1), 1-12.
- Fuchs, S. (2019). Vocal tract variations affect vowel sounds. Nature Human Behaviour, 3(10), 1043-1044.
- Żygis, M., & Fuchs, S. (2023). Communicative constraints affect oro-facial gestures and acoustics: Whispered vs normal speech. The Journal of the Acoustical Society of America, 153(1), 613–626. https://doi.org/10.1121/10.0015251
Dr. Aleksandra Ćwiek
(Principal Investigator)
Leibniz-Zentrum Allgemeine Sprachwissenschaft Berlin
cwiek@leibniz-zas.de
Aleksandra Ćwiek is a researcher at the Leibniz-Centre General Linguistics (ZAS) in Berlin and a PI of the project “On the FLExibility and Stability of gesture-speecH coordination (FLESH): Evidence from production, comprehension, and imitation”. She received her Ph.D. from the Humboldt University of Berlin in 2022. In her doctoral thesis, she studied the diversity of linguistic iconicity and its role in language evolution and in language nowadays. She examined aspects such as iconic prosody, sound symbolism, and German ideophones. Her further interests include acoustic phonetics, speech-gesture link, cognition, and cross-linguistic research.
Selected publications
- Ćwiek, A. (2022). Iconicity in Language and Speech [PhD Thesis, Humboldt University of Berlin].
https://doi.org/10.18452/24544 - Ćwiek, A., Fuchs, S., Draxler, C., Asu, E. L., Dediu, D., … Perlman, M. (2021). Novel vocalizations are understood across cultures. Scientific Reports, 11(1), 10108. https://doi.org/10.1038/s41598-021-89445-4
- Ćwiek, A., Fuchs, S., Draxler, C., Asu, E. L., Dediu, D., … Winter, B. (2022). The bouba/kiki effect is robust across cultures and writing systems. Philosophical Transactions of the Royal Society B: Biological Sciences, 377(1841), 20200390.
- Fuchs, S., & Ćwiek, A. (2022). Sounds Full of Meaning and the Evolution of Language. Acoustics Today, 18(1), 43–51. https://doi.org/10.1121/AT.2022.18.2.43
- Wagner, P., Ćwiek, A., & Samlowski, B. (2019). Exploiting the speech-gesture link to capture fine-grained prosodic prominence impressions and listening strategies. Journal of Phonetics, 76, 100911.
https://doi.org/10.1016/j.wocn.2019.07.001
Dr. Wim Pouw
(Principal Investigator)
Donders Institute &
MPI for Psycholinguistics Nijmegen
w.pouw@donders.ru.nl
(A short introduction and selected publications are to be updated.)
Šárka Kadavá
(PhD Candidate)
Leibniz-Zentrum Allgemeine Sprachwissenschaft Berlin
kadava@leibniz-zas.de
Šárka Kadavá is a doctoral researcher in the DFG project On the FLExibility and Stability of gesture-speecH coordination (FLESH): Evidence from production, comprehension, and imitation. She is currently based in Leibniz-Zentrum Allgemeine Sprachwissenschaft (ZAS) in Berlin. Her research focuses on multimodality, on how different modalities contribute to the sharing of meaning and how they are coordinated with each other, and on how language-like structural features emerge in non-normative communication systems. She has also participated in a project developing Czech adaptation of MB-CDI at the Institute of Psychology, Czech Academy of Sciences.
Project Description
Speech and gestures involve an orchestra of motions working in concert for communicative targets. Several previous studies have shown that speech and gestures are coordinated. However, it is less clear how stable or flexible coordination is and how their coordination evolves. In the FLESH project, we assume that the coordination between speech and gesture is related to the meaning, and plays an important role in the semiotic process. The originality of our approach is that we take the motor properties into account. We suppose that stable gesture-vocal coordination needs to integrate the lighter mass and higher velocities of speech articulators in comparison to arm gestures. Moreover, in contrast to many previous theoretical approaches, we see coordination between speech and gesture as crucial for language evolution.
To date, gesture studies and language evolution research did not consider motor control in detail and focused on either vocalization or gesture, thus leaving out their coordination and multimodality in general.
In the FLESH project, we aim to change this by studying the coordination of vocalizations and gestures in the construction of meaning. FLESH is organized in four work packages. In work package (WP) 1, we assess the natural coordination tendencies of gesture and vocalization separately and in cooperation when expressing meanings. We assess which coordination tendencies are actually relevant for understanding the meaning of gesture-vocal utterances.
In WP 2, state of the art technology and signal processing methodologies will be used to investigate the coordination of the orchestra of multiple motions during imitation. With a high level of granularity, we will be able to study several gesture-vocal articulators that are involved, to see whether particular types of articulators are more likely to coordinate in meaningful versus non-meaningful imitations. In WP 3 we assess how gesture-vocal utterances evolve when they are passed on and imitated by a connected chain of imitators. This experiment will simulate language evolution in the lab. The focus will be on how coordination tendencies change or increase with repetition of gesture-vocal utterances from one chain to the next. In WP 4 we will artificially change the temporal coordination between vocalization and gesture in order to see whether this affects semantic processing. Additionally, gesture-vocal utterances will be presented that do not match in terms of their meaning to see whether novel meanings emerge. The FLESH-project will have an impact on the theoretical understanding of gesture-speech relations, the relation between meaning and motor coordination, and its role in language evolution.
Project Activities
Publications
Bauer, A., Trettenbrein, P. C., Amici, F., Ćwiek, A., Fuchs, S., Krause, L., Kuder, A., Ladewig, S., Schulder, M., Schumacher, P., Spruijt, D., Zulberti, C. & Schulte-Rüther, M. (2025). Data Collection in Multimodal Language and Communication Research: A Flexible Decision Framework. https://doi.org/10.31234/osf.io/42tud_v1
Ćwiek, A., Anselme, R., Dediu, D., Fuchs, S., Kawahara, S., Oh, G., Paul, J., Perlman, M., Caterina, P., Reiter, S., Ridouane, R., Zeller, J., & Winter, B. (2024). The alveolar trill is perceived as jagged/rough by speakers of different languages. The Journal of the Acoustical Society of America. https://doi.org/10.1121/10.0034416
Eigsti, I. M., & Pouw, W. (2024). Deliberate synchronization of speech and gesture: effects of neurodiversity and development. Language and Cognition, 16(4), 1812-1833.
Fuchs, S., Kadavá, Š., Walker, B., Nicolas, F. & Winter, B., Ćwiek, A. (2024) Exploring the sound structure of novel vocalizations. Proceedings of the 15th International Conference on the Evolution of Language (EvoLang XV)
Jenkins, T., & Pouw, W. (2023). Gesture–speech coupling in persons with aphasia: A kinematic-acoustic analysis. Journal of Experimental Psychology: General, 152(5), 1469.
Kadavá, Š., Ćwiek, A., Fuchs, S. & Pouw, W. (2023) Is gesture-speech physics at work in rhythmic pointing? Evidence from Polish counting-out rhymes. Proceedings of the 20th International Congress of Phonetic Sciences (ICPhS), Prague, 7-8 August 2023
Kadavá, Š., Cwiek, A., Fuchs, S., & Pouw, W. (2023). What do we mean when we say gestures are more expressive than vocalizations? An experimental and simulation study. In Proceedings of the Annual Meeting of the Cognitive Science Society (Vol. 46).
Kadavá, S. K., Cwiek, A., Stoltmann, K., Fuchs, S., & Pouw, W. (2023). Is gesture-speech physics at work in rhythmic pointing? Evidence from Polish counting-out rhymes. Proceedings of the 20th International Congress of Phonetic Sciences.
Kadavá, Š. & Pouw, W. (2024) 3D motion tracking using multiple cameras with OpenPose and Pose2Sim. Script
Kadavá, Š., Snelder, J. & Pouw, W. (2024) Recording from Multiple Webcams Synchronously while LSL Streaming. Script
Kilpatrick, A. J., & Ćwiek, A. (2024). Using artificial intelligence to explore sound symbolic expressions of gender in American English. PeerJ Computer Science, 10, e1811. https://doi.org/10.7717/peerj-cs.1811 
Kilpatrick, A. J.*, Ćwiek, A.*, & Kawahara, S. (2023). Random forests, sound symbolism and Pokémon evolution. PLOS One, 18(1), e0279350. DOI: 10.1371/journal.pone.0279350 (* gemeinsame Erstautor*innenschaft)
Kilpatrick, A., Ćwiek, A., Lewis, E., & Kawahara, S. (2023). A cross-linguistic, sound symbolic relationship between labial consonants, voiced plosives, and Pokémon friendship. Frontiers in Psychology, 14, 722. DOI: 10.3389/fpsyg.2023.1113143
Nalbantoğlu, H. & Kadavá, Š. (2025) Multi-Scenario Video and Audio Synchronization and Segmentation. Script
Ngai, C. H., Kilpatrick, A. J., & Ćwiek, A. (2024). Sound symbolism in Japanese names: Machine learning approaches to gender classification. PLOS ONE, 19(3), e0297440. https://doi.org/10.1371/journal.pone.0297440
Owoyele, B. A., Schilling, M., Sawahn, R., Kaemer, N., Zherebenkov, P., Verma, B., Pouw, W. & de Melo, G. (2024). MaskAnyone Toolkit: Offering Strategies for Minimizing Privacy Risks and Maximizing Utility in Audio-Visual Data Archiving. Proceedings of the 2024 International Conference on Information Systems.
Pouw, W. (preprint). Icons in Action: Redefining Iconicity for the Cognitive Sciences.
Pearson, L., Nuttall, T., & Pouw, W. (prepint). Landscapes of coarticulation: The co-structuring of gesture-vocal dynamics in Karnatak vocal performance.
Pouw, W., Yung, B., Shaikh, S. A., Trujillo, J., Rueda-Toicen, A., de Melo, G., & Owoyele, B. (preprint). EnvisionHGdetector: A Computational Framework for Co-Speech Gesture Detection, Kinematic Analysis, and Interactive Visualization.
Pouw, W., Werner, R., Burchardt, L. S., & Selen, L. P. J. (2025). The human voice aligns with whole-body kinetics. Proceedings of the Royal Society B. https://doi.org/10.1098/rspb.2025.0160
Ćwiek, A., Kreiman, J., & Fuchs, S. (2025). Introduction to the Special Issue on Iconicity and Sound Symbolism. The Journal of the Acoustical Society of America, 157(5), 3806–3813. https://doi.org/10.1121/10.0036662
Conference contributions
Henlein, A., Bauer, A., Bhattacharjee, R., Ćwiek, A., Gregori, A., Kügler, F., Lemanski, J., Lücking, A., Mehler, A., Prieto, P., Sánchez-Ramón, P. G., Schepens, J., Schulte-Rüther, M., Schweinberger, S. R., & von Eiff, C. I. (2024). An Outlook for AI Innovation in Multimodal Communication Research. In V. G. Duffy (Ed.), Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management (pp. 182–234). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-61066-0_13
Kadavá, Š., Ćwiek, A., Fuchs, S., & Pouw, W. (2024). What do we mean when we say gestures are more expressive than vocalizations? An experimental and simulation study. Proceedings of the Annual Meeting of the Cognitive Science Society, 46, 2308–2315. https://escholarship.org/uc/item/2mp1v3v5 (*)
Fuchs, S., Kadavá, Š., Pouw, W., Walker, B., Fay, N., Winter, B., & Ćwiek, A. (2024). Exploring the sound structure of novel vocalizations. Proceedings of EVOLANG 2024. EVOLANG XV, Madison, Wisconsin, USA. https://evolang2024.github.io/proceedings/paper.html?nr=113
Ćwiek, A., Kadavá, Š., Stoltmann, K., Pouw, W., & Fuchs, S. (2023). When Gestures Enter the Game, Prosody Breaks the Rules. Abstract in Proceedings of GeSpIn. Gestures and Speech in Interaction, Nijmegen, Niederlande. URL (*)
Kadavá, Š., Ćwiek, A., Stoltmann, K., Fuchs, S., & Pouw, W. (2023). Is gesture-speech physics at work in rhythmic pointing? Evidence from Polish counting-out rhymes. Proceedings of the 20th International Congress of Phonetic Sciences. ICPhS 2023, Prag, Tschechien. DOI: 10.31219/osf.io/67fzc (*)
Żygis, M., Błaszczak, J., Ćwiek, A., Saint-Pettersen, M., Wesołek, S., & Gulgowski, P. (2023). Attitudes in the German-Polish Context Based on Perceptual Evidence. Proceedings of the 20th International Congress of Phonetic Sciences. ICPhS 2023, Prag, Tschechien. URL (*)
Gregori, A., Amici, F., Brilmayer, I., Ćwiek, A., Fritzsche, L., Fuchs, S., Henlein, A., Herbort, O., Kügler, F., Lemanski, J., Liebal, K., Lücking, A., Mehler, A., Nguyen, K. T., Pouw, W., Prieto, P., Rohrer, P. L., Sánchez-Ramón, P. G., Schulte-Rüther, M., Schumacher, P., Schweinberger, S., Struckmeier, V., Trettenbein, P., & von Eiff, C. (2023). A Roadmap for Technological Innovation in Multimodal Communication Research. In: Duffy, V.G. (eds) Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. HCII 2023. Lecture Notes in Computer Science, vol 14029, 402-438. Springer, Cham. DOI: 10.1007/978-3-031-35748-0_30 (*)
Christoph, N., & Ćwiek, A. (2022). Lautsymbolische Größencodierung bei der Benennung von Hunden. Abstract in Tagungsband der 18. Tagung Phonetik und Phonologie im Deutschsprachigen Raum, 27–29, Bielefeld, Deutschland. DOI: 10.11576/pundp2022-1032
Invited talks and guest lectures
Ćwiek, A. (11.2024). The RISE of a PhD: Navigating Chaos, Crisis, and Success in Academia. Internal PhD Workshop, ZAS Berlin, Deutschland.
Ćwiek, A. (11.2024). What beats the bouba-kiki effect? CLES Kolloquium, Toruń, Polen.
Ćwiek, A. (11.2024). Some preliminary ideas on the natural statistics of uni- and multimodal novel utterances. CLES Kolloquium, Toruń, Polen.
Ćwiek, A. (09.2024). The RISE of a PhD: Navigating Chaos, Crisis, and Success in Academia. LinPin 2024, Aachen, Deutschland.
Ćwiek, A., & Fuchs, S. (01.2024). The Coordination of Dynamic Multimodal Signals in Novel Communication. Roundtable – Multimodal Speech Data. Data – Annotation – Strategies of Analysis, Goethe-Universität Frankfurt, Deutschland.
Ćwiek, A. (01.2024). Exploring Gesture-Speech Dynamics. Kolloquium der Arbeitsgruppe Phonetik, Universität Bielefeld, Deutschland.
Ćwiek, A. (09.2023). Exploring the Dynamic Interplay of Gestures and Prosody in Polish Counting-Out Rhymes. Seminar der Gruppe für Prosodie- und Gestenstudien (GrEP-G), Universitat Pompeu Fabra, Barcelona, Spanien.
Ćwiek, A. (01.2023). Lautsymbolik um uns herum. Seminar „Sprachproduktion und -perzeption“, Humboldt-Universität zu Berlin, Deutschland.
Ćwiek, A. (10.2022). Language and motion. Workshop zu Labormethoden in der Linguistik, Leibniz-Zentrum Allgemeine Sprachwissenschaft, Berlin, Deutschland
Fuchs, S. (09.2024) Moving while talking. Invited ILCB welcome back party – ILCB fête de la rentrée. ILCB, Université d’Aix-Marseille, France.
Fuchs, S. (08.2024) Interdisciplinary approaches to respiratory rhythms. Keynote speaker at ILCB Summer school, Marseille, France.
Fuchs, S. (10.2023) A comparative approach to communicative alignment and adaptation from nonhuman animals. (Mis)Alignment Workshop, Laboratoire Parole et Langage, Aix-en-Provence, France.
Fuchs, S. (09.2023) The role of bones, joints and muscles for speech and gesture in interaction. Keynote speaker at Gestures in Speech and Interaction (Gespin), Max Planck Institute for Psycholinguistics, Nijmegen.
Kadavá, Š. (12.2024) To interact is to move: movement as a core constituent of social interaction. Multimodality in Social Interaction workshop, Marseille, France.
Teaching
Summer and winter school courses
Ćwiek, A. Iconicity in Communication. DGfS Summerschool “Form-Meaning Mismatches in Spoken and Visual Communication”, Göttingen, August 12 – 16, 2024.
Kadavá, Š. Visualizing Multimodal Data. LOT Winter School 2024, Nijmegen, January 13 – 17, 2025.
Pouw, W., Owoyele, B. Reproducible Computational Multimodal Analysis. LOT Winter School 2024, Nijmegen, January 13 – 17, 2025.
Pouw, W., Owoyele, B. Quantitative and computational approaches of gestures. DGfS Summerschool “Form-Meaning Mismatches in Spoken and Visual Communication”, Göttingen, August 12 – 16, 2024.
Events
| Date | Event | Organization |
| 15.10.2024 | Susanne Fuchs: Atmung, Stimme und Bewegung. Book a scientist, Leibniz Association | |
| 12.09.2024 | The 21st International Congress of Linguists (ICL), Workshop “All Shades of Iconicity: Ideophones, Onomatopoeia, and Sound Symbolism” | Aleksandra Ćwiek, Maria Flaksman & Kathryn Barnes |
| 22.06.2024 | FLESH at the Lange Nacht der Wissenschaften, Berlin Leibniz Headquarter | |
| 06.2024 – 01.2025 | Gillian Rosenberg (Princeton University) Internship within FLESH at ZAS | |
| 02.-04.05.2024 | Workshop Behavioral Dynamics in Social Interactions, Krakow | James Trujillo, Arkadiusz Bialłek & Wim Pouw |
| 25.04.2024 | Invited talk by Bob Ladd, ZAS Berlin | Aleksandra Ćwiek |
| 11.-16.03.2024 | FLESH at InScience Festival, Nijmegen | |
| 16.11.2023 | Workshop “The practical application of decision tree-based AI models” von Alexander J. Kilpatrick, ZAS Berlin | Aleksandra Ćwiek |
| 21.07.2023 | 45th Annual conference of the German Linguistic Society (Deutsche Gesellschaft für Sprachwissenschaft, DGfS), AG15 Ideophones and lexicalized iconicity in language | Kathryn Barnes & Aleksandra Ćwiek |
Short-term Collaborations
Testing the correlation between top-down prosodic annotation systems and bottom-up automatic annotation (2023, 2024) – Aleksandra Ćwiek, Pilar Prieto, Frank Kügler & Patrick Rohrer
Moving Meetings by Moving Prosody and Gesture (2024) – Alina Gregori & Susanne Fuchs
Developing computer vision tools for animal motion tracking (2025) – Šárka Kadavá & Chiara Zulberti
Kinematic Trajectory from Gesture to Sign (2024) – Šárka Kadavá & Door Spruijt
Thesis (in progress):
Kadavá, Š. (in prep.) Emergence of Meaning in Gesture-vocal Signaling. University of Göttingen and Radboud Universität Nijmegen.