Participants
(Participants are listed in alphabetical order by their last name.)
A – D
Dr. Federica Amici
(Principal Investigator)
Project:
Compositional structures
in chimpanzee gestural communication
University of Leipzig
amici@eva.mpg.de
Federica Amici obtained her PhD at the Liverpool John Moores University in the UK. She then moved to Germany, where she is currently working in Katja Liebal’s group as a post-doc. Her main research interests lie in the evolutionary processes shaping the distribution of behavior and cognition in animals, with a special focus on primates and ungulates. Together with Katja Liebal, I am currently working on the complexity and ontogeny of primate communication, and I maintain a secret but intense love for psycholinguistics.
Selected publications
- Amici, F., Lembeck, M., Holodynski, M., & Liebal, K. 2022. Face to face interactions in chimpanzee and human mother-infant-dyads. Philosophical Transactions B.
- Amici, F., & Liebal, K. 2022. Primates unleashed. Behavioral and Brain Sciences.
- Amici, F., & Liebal, K. 2022. The social dynamics of communication in great and lesser apes (Pan troglodytes, Pongo pygmaeus abelii, Symphalangus syndactylus). Philosophical Transactions B, 377, 20210299.
- Amici, F., Oña, L., & Liebal, K. 2022. Compositionality in primate gestural communication and multicomponent signal displays. International Journal of Primatology. https://doi.org/10.1007/s10764-022-00316-9.
- Amici, F., Sanchez-Amaro, A., & Cacchione, T. 2022. Branching and working memory: a cross-linguistic approach. In: The Cambridge handbook of working memory and language (Eds. Schwiete, J.W., & Wen, Z.). Cambridge University Press.
Dr. Mailin Ines Antomo
(Principal Investigator)
Project:
Lying, deceiving, misleading:
are we committed to our gestures?
University of Göttingen
mailin.antomo@phil.uni-goettingen.de
Mailin Antomo’s research investigates phenomena at the interface of syntax, semantics, and pragmatics with a focus on German. She has worked, both theoretically and empirically, on embedded root phenomena, (not-)at-issueness, implicatures, presuppositions, lying and deceiving, and the acquisition of these phenomena. Furthermore, she is interested in educational linguistics. She obtained her PhD from the University of Göttingen in 2015, after being part of the DFG-funded research training group “Sentence types: Variation and Interpretation” at the university of Frankfurt. She is currently a permanent lecturer for German linguistics at the University of Göttingen and principal investigator of the project “Lying, deceiving, misleading: are we committed to our gestures?”.
Selected publications
- Antomo, Mailin (to appear): Lying with visual means. To appear in: Wiegmann, Alex (ed.): Lying, Fake News, and Bullshit. Bloomsbury, Advances in Experimental Philosophy.
- Chen, Yuqiu/Thalmann, Maik/Antomo, Mailin (2022): Presuppositions and at-issueness: Insights from language acquisition into the soft-hard distinction. In: Journal of Pragmatics 199, 21-46.
- Thalmann, Maik/Chen, Yuqiu/Müller, Susanne/Paluch, Markus/Antomo, Mailin (2021): Against PCI-GCI uniformity: Evidence from deceptive language in German and Chinese. In: Linguistische Berichte 267, 355-385.
- Antomo, Mailin/Müller, Susanne/Paluch, Markus/Paul, Katharina/Thalmann, Maik (2018): When children aren’t more logical than adults: An empirical investigation of lying by falsely implicating. In: Journal of Pragmatics 138, 135-148.
- Antomo, Mailin (2012): Projective Meaning and the Licensing of Embedded Root Phenomena. In: Boone, Enrico/Linke, Kathrin/Schulpen, Maartje (ed.): Proceedings of ConSOLE XIX, Leiden, 1-23.
Dr. Nadine Bade
(Principal Investigator)
Project:
Pragmatic reasoning with
(non-)visual alternatives
University of Potsdam
nadine.bade@uni-potsdam.de
Nadine Bade’s research focuses on experimental investigations of phenomena at the semantics-pragmatics interface. Her work addresses the question how different contextual and structural factors influence pragmatic inferences and reasoning. Her research mostly uses methods from psychology to study pragmatic meaning in a variety of languages, including but not limited to mouse- and eye-tracking, reaction time as well as rating studies. Before starting her project “Pragmatic reasoning with (non-)visual alternatives” in ViCom she was working in different interdisciplinary projects at the University of Potsdam, École Normale Supérierure in Paris and the University of Tübingen.
Selected publications
- Bade, Nadine, et al. “Alternatives and attention in language and reasoning: A reply to Mascarenhas & Picat 2019.” Semantics and Pragmatics 15.2 (2022).
- Bade, Nadine, and Florian Schwarz. “New data on the competition between definites and indefinites.” Experiments in Linguistic Meaning 1 (2021): 15-26.
- Bade, Nadine. “On the scope and nature of Maximise Presupposition.” Language and Linguistics Compass 15.6 (2021): e12416.
- Bade, Nadine, and Agata Renans. “A cross-linguistic view on the obligatory insertion of additive particles—Maximize Presupposition vs. Obligatory Implicatures.” Glossa: a journal of general linguistics 6.1 (2021).
- Bade, Nadine, and Konstantin Sachs. “EXH passes on alternatives: A comment on Fox and Spector (2018).” Natural Language Semantics 27.1 (2019): 19-45.
Dr. Anastasia Bauer
(Principal Investigator)
University of Cologne
anastasia.bauer@uni-koeln.de
Anastasia Bauer is interested in different anguage modalities and in contact between them. Her current work is focused on comparing constructions sharing a similar form in co-speech gesture and sign language. As a PhD fellow in the project „Village Sign“ within the collaborative research programme (EUROCORES) EuroBABEL „Better Analyses Based on Endangered Languages“, she worked on an endangered sign language in Aboriginal Australia. Her dissertation explores the spatial grammar in Yolŋu sign language. As a postdoc, she worked at the universities of Hamburg and Cologne. Her postdoctoral project focuses on two cross-modal language contact phenomena in Russian Sign Language, i.e. mouthings and fingerspellings. She is currently the Principal Investigator of the ViCom project “Gestures or signs? Comparing manual and non-manual constructions sharing the same form in co-speech gesture and sign language: a corpus-driven approach. (GeSi)”.
Selected publications
- Mohr, Susanne & Anastasia Bauer (2022). Gesture, sign languages and multimodality. Svenja Völkel & Nico Nassenstein (Eds.). Approaches to Language and Culture, De Gruyter Mouton, 159-196. https://doi.org/10.1515/9783110726626-007
- Bauer, Anastasia & Maria Kyuseva. (2022) New insights on mouthings in sign languages: evidence from corpus and elicitation studies of Russian sign language. Frontiers in Psychology. https://doi.org/10.3389/fpsyg.2021.779958
- Bauer, Anastasia. 2020. Das Konzept der multimodalen Sprache am Beispiel von der Russischen Gebärdensprache. Bulletin der deutschen Slavistik 26, Berlin: Frank & Timme Verlag, 131-139.
- Bauer, Anastasia. 2019. „When words meet signs: A corpus-based study on variation of mouthing in Russian Sign Language“. In: Anastasia Bauer & Bunčić, Daniel (Eds.). Linguistische Beiträge zur Slavistik, Specimina philologiae Slavicae 198, pp. 9-35.
- Bauer, Anastasia. 2014. The Use of Signing Space in a Shared Sign Language of Australia. Berlin: Mouton de Gruyter. https://doi.org/10.1515/9781614515470
Dr. Reetu Bhattacharjee
(Postdoc Researcher)
Project:
Gestures and Diagrams in Visual-Spatial Communication: Methodological Tools and Applications in Mathematics and Logic
University of Münster
reetu.bhattacharjee@uni-muenster.de
(Photo, a short introduction, and selected publications are to be updated.)
Yuqiu Chen
(Postdoc Researcher)
Project:
Lying, deceiving, misleading: are we committed to our gestures? in Mathematics and Logic
University of Göttingen
yuqiu.chen@uni-goettingen.de
Yuqiu Chen is mainly interested in phenomena at the semantics-pragmatics interface and has worked, amongst others, on presuppositions, at-issueness, implicatures with a focus on their acquisition and cross-linguistic comparison. Recently, Yuqiu has defended her PhD thesis“Presuppositions at the Semantics-Pragmatics Interface: Experimental Studies on Their Classification, Acquisition and Cross-Linguistic Comparison”. She will join the project “Lying, deceiving, misleading: are we committed to our gestures?” as a post-doc in 2024.
Selected publications
- Yuqiu Chen, Maik Thalmann, Mailin Antomo (2022): “Presupposition triggers and (not-) at-issueness: Insights from language acquisition into the soft-hard distinction” In: Journal of Pragmatics (199), 21-46
- Yuqiu Chen (2022): “Wenn Nicht-Muttersprachler/innen mehr Toleranz zeigen: Eine experimentelle Studie zu Präsuppositionen, At-issueness und DaF”, In: Auteri, Laura et al. (Hrsg.): Wege der Germanistik in transkultureller Perspektive (Bd. 6). Jahrbuch für Internationale Germanistik Beihefte, 213-227.
- Maik Thalmann, Yuqiu Chen, Susanne Müller, Markus Paluch, Mailin Antomo (2021): “Against PCI-GCI uniformity. Evidence from deceptive language in German and Chinese.” In: Linguistische Berichte, 235-285.
- Yuqiu Chen (2019): “An experimental study on scalar implicatures: Comparing German native speakers and Chinese learners of German.” In: Proceedings of ConSOLE XXVI, 116–138.
Dr. Aleksandra Ćwiek
(ViCom Data Network Coordinator, PI)
Leibniz-Zentrum Allgemeine Sprachwissenschaft Berlin
cwiek@leibniz-zas.de
Aleksandra Ćwiek is a researcher at the Leibniz-Centre General Linguistics (ZAS) in Berlin and a PI of the project “On the FLExibility and Stability of gesture-speecH coordination (FLESH): Evidence from production, comprehension, and imitation”. She received her Ph.D. from the Humboldt University of Berlin in 2022. In her doctoral thesis, she studied the diversity of linguistic iconicity and its role in language evolution and in language nowadays. She examined aspects such as iconic prosody, sound symbolism, and German ideophones. Her further interests include acoustic phonetics, speech-gesture link, cognition, and cross-linguistic research.
Selected publications
- Ćwiek, A. (2022). Iconicity in Language and Speech [PhD Thesis, Humboldt University of Berlin].
https://doi.org/10.18452/24544 - Ćwiek, A., Fuchs, S., Draxler, C., Asu, E. L., Dediu, D., … Perlman, M. (2021). Novel vocalizations are understood across cultures. Scientific Reports, 11(1), 10108. https://doi.org/10.1038/s41598-021-89445-4
- Ćwiek, A., Fuchs, S., Draxler, C., Asu, E. L., Dediu, D., … Winter, B. (2022). The bouba/kiki effect is robust across cultures and writing systems. Philosophical Transactions of the Royal Society B: Biological Sciences, 377(1841), 20200390.
- Fuchs, S., & Ćwiek, A. (2022). Sounds Full of Meaning and the Evolution of Language. Acoustics Today, 18(1), 43–51. https://doi.org/10.1121/AT.2022.18.2.43
- Wagner, P., Ćwiek, A., & Samlowski, B. (2019). Exploiting the speech-gesture link to capture fine-grained prosodic prominence impressions and listening strategies. Journal of Phonetics, 76, 100911.
https://doi.org/10.1016/j.wocn.2019.07.001
Prof. Dr. Christian Dobel
(Principal Investigator)
Project:
Audiovisual Perception of Emotion and Speech
in Hearing Individuals and Cochlear Implant Users
Jena University Hospital
christian.dobel@med.uni-jena.de
Christian Dobel is a Professor of Experimental Otorhinolaryngology at the Jena University Hospital.
Selected publications
- Dobel C, Diesendruck G, Bölte J. How writing system and age influence spatial representations of actions: a developmental, cross-linguistic study. Psychol Sci. 2007 Jun;18(6):487-91. doi: 10.1111/j.1467-9280.2007.01926.x. PMID: 17576259.
- Dobel C, Enriquez-Geppert S, Hummert M, Zwitserlood P, Bölte J. Conceptual representation of actions in sign language. J Deaf Stud Deaf Educ. 2011 Summer;16(3):392-400. doi: 10.1093/deafed/enq070. Epub 2011 Feb 21. PMID: 21339342.
- Dobel C, Enriquez-Geppert S, Zwitserlood P, Bölte J. Literacy shapes thought: the case of event representation in different cultures. Front Psychol. 2014 Apr 16;5:290. doi: 10.3389/fpsyg.2014.00290. PMID: 24795665; PMCID: PMC3997043.
- Dobel C, Nestler-Collatz B, Guntinas-Lichius O, Schweinberger SR, Zäske R. Deaf signers outperform hearing non-signers in recognizing happy facial expressions. Psychol Res. 2020 Sep;84(6):1485-1494. doi: 10.1007/s00426-019-01160-y. Epub 2019 Mar 13. PMID: 30864002.
- Roesmann K, Dellert T, Junghoefer M, Kissler J, Zwitserlood P, Zwanzger P, Dobel C. The causal role of prefrontal hemispheric asymmetry in valence processing of words – Insights from a combined cTBS-MEG study. Neuroimage. 2019 May 1;191:367-379. doi: 10.1016/j.neuroimage.2019.01.057. Epub 2019 Feb 1. PMID: 30716460.
E – G
Prof. Dr. Cornelia Ebert
(Coordinator, Steering Committee Member, PI)
Project:
Visual and Non-visual Means of
Perspective Taking in Language
Goethe University Frankfurt
ebert@lingua.uni-frankfurt.de
Cornelia Ebert is a professor of linguistics/semantics at Goethe University in Frankfurt am Main. She received her Ph.D. in lingusitics at Potsdam University in 2006. Her thesis was titled »Quantificational topics. A scopal treatment of exceptional wide scope phenomena«. She was a lecturer and a researcher at Osnabrück University, Stuttgart University and the Leibniz-Zentrum Allgemeine Sprachwissenschaft in Berlin. Since 2021 she carries out a British-German cooperative research project entitled »Interactions between Dynamic Effects and Alternative-Based Inferences in the Study of Meaning« (IDEALISM). And since 2022 she is one of the coordinators of the DFG Priority Programme 2392 Visual Communication. Theoretical, Empirical, and Applied Perspectives (ViCom). 2020-24 she is Goethe Fellow at the Forschungskolleg Humanwissenschaften.
Selected publications
- Barnes, K. R., Ebert, C., Hörnig, R., & Stender, T. (2022). The at-issue status of ideophones in German: An experimental approach. Glossa: A Journal of General Linguistics, 7(1). https://doi.org/10.16995/glossa.5827
- Ebert, C., Ebert, C., & Hörnig, R. (2020). Demonstratives as dimension shifters. Proceedings of Sinn Und Bedeutung, 161-178 Pages. https://doi.org/10.18148/SUB/2020.V24I1.859
- Ebert, C. (2018). A comparison of sign language with speech plus gesture. Theoretical Linguistics, 44(3–4), 239–249. https://doi.org/10.1515/tl-2018-0016
- Ebert, C., Ebert, C., & Hinterwimmer, S. (2014). A unified analysis of conditionals as topics. Linguistics and Philosophy, 37(5), 353–408. https://doi.org/10.1007/s10988-014-9158-4
- Ebert, C. (2009). Quantificational topics: A scopal treatment of exceptional wide scope phenomena (Vol. 86). Springer Science & Business Media.
Prof. Dr. Natalia Filatkina
(Principal Investigator)
University of Hamburg
natalia.filatkina@uni-hamburg.de
Natalia Filatkina is a professor for German linguistics at Universität Hamburg with a focus on digital historical linguistics. Her research focuses on historic linguistics of German, language change, and multilingualism, amongst others. She received her PhD from Otto Friedrich Universität Bamberg in 2003. After being a scientific assistant (C1) at the Universität Trier, she was Director of the junior researcher group “Historische Formelhafte Sprache und Traditionen des Formulierens (HiFoS)” at the Universität Trier. Since then she has been a deputy professor at the Heinrich Heine-Universität Düsseldorf, academy professor at the Universität Trier and at the same time at the Akademie der Wissenschaften und der Literatur in Mainz. She obtained her habilitation at the Universität Trier. Since 2021, she is president of the Europhras-Gesellschaft.
Selected publications
- Filatkina, N. 2018. Historische formelhafte Sprache. Theoretische Grundlagen und Methoden ihrer Erforschung [Habilitationsschrift]. Berlin & Boston: De Gruyter Mouton.
- Filatkina, N., C. Moulin, I. Gurevych & R. Eckart de Castilho. 2015. Analyzing Formulaic Patterns in Historical Corpora. In Gippert, J. & R. Gehrke (eds.), Historical Corpora: Challenges and perspectives. Tübingen: Narr, 51–63.
- Filatkina, N. 2014. Constructionalization, Konstruktionswandel und figurative Sprache (sprach)historisch betrachtet. In Dalmas, M. & E. Piirainen (eds.), Figurative Sprache – Figurative Language – Langage figuré. Festgabe für Dmitrij O. Dobrovol’skij. Tübingen: Stauffenburg Verlag, 41–57.
- Filatkina, N. 2010. Phraseologie der germanischen Sprachen kontrastiv: Geschichte, Ergebnisse und Perspektiven. In Dammel, A., S. Kürschner & D. Nübling (eds.), Kontrastive Germanische Linguistik. Teilband 1 [Germanistische Linguistik, 206–209]. Hildesheim: Olms Weidmann, 275–309.
- Filatkina, N. 2005. Multi-methodologische Korpuserstellung als empirische Basis für phraseologische und phraseographische Untersuchungen. Am Beispiel des Lëtzebuergeschen. In Eckhard, E., J. E. Schmidt & D. Stellmacher (eds.), Moderne Dialektologie – Neue Dialektologie. Akten des 1. Kongresses der Internationalen Gesellschaft für Dialektologie des Deutschen (IGDD) am Forschungsinstitut für deutsche Sprache „Deutscher Sprachatlas“ der Philipps-Universität Marburg vom 5.-8. März 2003 [Zeitschrift für Dialektologie und Linguistik – Beihefte, vol. 130]. München: Franz Steiner, 555–572.
Thomas Finkbeiner
(Principal Investigator)
Project:
Parts of Speech and Iconicity in
German Sign Language (DGS)
University of Göttingen
thomas.finkbeiner@uni-goettingen.de
Thomas Albert Finkbeiner is deaf and grew up with German Sign Language (DGS) as his native language. He is a certified social worker / certified social pedagogue (FH), state-certified sign language lecturer and state-certified sign language interpreter for DGS and International Sign. Since 2017, he has been working at the Seminar for German Philology at the Georg-August-University Göttingen as a lecturer for DGS and Deaf Studies. He is involved in various research on sign languages as a researcher, consultant and translator. He is editing the first bimodal-bilingual book series in German Sign Language (DGS) and written German ‘Deutsche Gebärdensprache und Deaf Communities’ (Buske Verlag, with Nina-Kristin Meister) and the ‘German Sign Language Calendar’ (Buske Verlag, with Nina-Kristin Meister).
Selected publications
- Thomas Finkbeiner, Nina-Kristin Meister and Alexander Silbersdorff (in prep.): Gemeinsam auf dem Weg zur barrierefreien digitalen Hochschullehre. Mathematische Lehrvideos mit Deutscher Gebärdensprache (DGS) (in prep.). In: Neues Handbuch Hochschullehre. Berlin: DUZ Verlags- und Medienhaus.
- Thomas Finkbeiner and Nina-Kristin Meister (in prep.): Sprachkalender Deutsche Gebärdensprache 2024. Hamburg: Buske.
- Thomas Finkbeiner, Liona Paulus and Nina-Kristin Meister (in prep.): 100 Fragen und Antworten rund um die Deutsche Gebärdensprache (DGS). Hamburg: Buske.
- Nina-Kristin Pendzich, Jens-Michael Cramer, Thomas Finkbeiner, Annika Herrmann and Markus Steinbach (2022): How do Signers Mark Conditionals in German Sign Language? Insights from a Sentence Reproduction Task on the Use of Nonmanual and Manual Markers. In: Croatian Review of Rehabilitation Research 58 (Special Issue: Sign Language, Deaf Culture, and Bilingual Education), 206-226.
- Thomas Finkbeiner and Nina-Kristin Pendzich (2022): Sprachkalender Deutsche Gebärdensprache 2023. Hamburg: Buske.
Lea Fricke
(Postdoc Researcher)
Project:
Semantics and Pragmatics of Emojis in
Digital Communication (EmDiCom)
University of Bochum
Lea.Fricke@ruhr-uni-bochum.de
Lea Fricke’s research interests lie in the area of semantics, pragmatics and empirical linguistics. She has worked on embedded questions, the German prefield-es, scalar implicatures and experimental methodology. In November 2022, Lea joined the Ruhr-University Bochum to work in the ViCom project “Semantics and Pragmatics of Emojis in Digital Communication”. Before that, she worked as a university assistant at the University of Graz and was part of the XPrag.de project “Exhaustiveness in embedded questions across languages“.
Selected publications
- Cremers, Alexandre, Lea Fricke, Edgar Onea. Accepted. The importance of being earnest: How truth and evidence affect participants’ judgments. Glossa Psycholinguistics.
- Cortez Espinoza, Maya, Lea Fricke. To appear. On the interpretation of German einige. The effect of tense and cardinality. Proceedings of ELM 2.
- Zimmermann, Malte, Lea Fricke, Edgar Onea. 2022. Embedded questions are exhaustive alright, but… In A. Özgün, Y. Zinova (eds.) Proceedings of TbiLLC 2019, 173–194. Cham: Springer.
- Fricke, Lea. 2020. A southern German use of prefield-es: Evidence from the corpus and an experimental study. Zeitschrift für Sprachwissenschaft 39(1), 41–77.
- Tönnis, Swantje, Lea M. Fricke, Alexander Schreiber. 2018. Methodological Considerations on Testing Argument Asymmetry in German Cleft Sentences. In E. Fuß, M. Konopka, B. Trawiński, U. H. Waßner (eds.). Grammar and Corpora 2016, 231–240. Heidelberg: Heidelberg University Publishing.
Dr. Susanne Fuchs
(ViCom Data Network Coordinator, Steering Committee Member, PI)
Leibniz-Zentrum Allgemeine Sprachwissenschaft Berlin
fuchs@leibniz-zas.de
Susanne Fuchs (ZAS Berlin) investigates the biopsychosocial foundations of human interaction and focuses specifically on physiological processes, such as breathing and motor control. Her main areas of interests are:
1) The interplay between motion, breathing and cognition,
2) Speech preparation and pauses,
3) Multimodality and iconicity,
4) Biological and social aspects shaping individual behaviour in speech production and perception.
She uses manifold techniques, among them optitrack, inductance plethysmography, electropalatography and intraoral pressure sensors.
Selected publications
- Pouw, W., & Fuchs, S. (2022). Origins of vocal-entangled gesture. Neuroscience & Biobehavioral Reviews, 104836.
- Fuchs, S., & Rochet-Capellan, A. (2021). The respiratory foundations of spoken language. Annual Review of Linguistics, 7(1), 13-30.
- Ćwiek, A., Fuchs, S., Draxler, C., Asu, E. L., Dediu, D., Hiovain, K., … & Perlman, M. (2021). Novel vocalizations are understood across cultures. Scientific Reports, 11(1), 1-12.
- Floegel, M., Fuchs, S., & Kell, C. A. (2020). Differential contributions of the two cerebral hemispheres to temporal and spectral speech feedback control. Nature communications, 11(1), 1-12.
- Fuchs, S. (2019). Vocal tract variations affect vowel sounds. Nature Human Behaviour, 3(10), 1043-1044.
- Żygis, M., & Fuchs, S. (2023). Communicative constraints affect oro-facial gestures and acoustics: Whispered vs normal speech. The Journal of the Acoustical Society of America, 153(1), 613–626. https://doi.org/10.1121/10.0015251
Prof. Dr. Volker Gast
(Principal Investigator)
Project:
Audiovisual Perception of Emotion and Speech
in Hearing Individuals and Cochlear Implant Users
University of Jena
volker.gast@uni-jena.de
(Photo, a short introduction, and selected publications are to be updated.)
Alina Gregori
(Steering Committee Member, PhD Candidate, PhD Contact Person)
Project:
Co-speech Gestures and Prosody as
Multimodal Markers of Information Structure
Goethe University Frankfurt
gregori@lingua.uni-frankfurt.de
Alina Gregori received her Bachelors (2021) and Masters (2022) degree in theoretical linguistics at Goethe University Frankfurt with a focus on Phonology. She started working as a PhD student in the project MultIS in October 2022. Within the project, she investigates the prosody-gesture link in communication and the impact of information structure (focus, topic, givenness) on the synchronization of gestures and prosodic entities in German. The bigger picture of the project includes a comparative analysis of German and Catalan with regard to the multimodal marking of information structure. A central aim in MultIS is the empirical approach to previously established prosody-gesture models, considering experimental as well as spontaneous speech utterances. Alina Gregoris MA thesis (title: “Co-speech Gestures, Information Structure and Prosody: A Corpus Study on Prominence Peak Alignment”) served as a pilot and preparation for MultIS.
Selected publications
- Henlein, A., A. Bauer, R. Bhattacharjee, A. Ćwiek, A. Gregori, F.Kügler, J. Lemanski, A. Lücking, A. Mehler, P. Prieto, P. G. Sánchez-Ramón, J. Schepens, M. Schulte-Rüther, S. R.Schweinberger & C. I. von Eiff. (2024). An Outlook for AI Innovation in MultimodalCommunication Research. Lecture Notes in Computer Science, Springer Cham.
- Gregori, A., P. G. Sánchez-Ramón, P. Prieto & F. Kügler. 2024. Prosodic and gestural marking of focus types in Catalan and German. In: Proc.Speech Prosody 2024, 891-895, doi: 10.21437/SpeechProsody.2024-180
- Kügler, F. & A. Gregori. (2023). Iconic Gestures in Focus – Synchronizationof Prosody and Gestures in Prominence. In: Skarnitzl, R. & Volín, J. (eds)Proceedings of 20th ICPhS 2023, Prague, Czech Republic August 2023. 4125-4129(ID 232). Guarant International.
- Gregori, A., F. Amici, I. Brilmayer, A. Ćwiek, L. Fritzsche, S. Fuchs, A. Henlein, O. Herbort, F. Kügler, J. Lemanski, K. Liebal, A.Lücking, A. Mehler, K.Tien Nguyen, W. Pouw, P. Prieto, P. L. Rohrer, P. G. Sánchez-Ramón, M. Schulte-Rüther, P. Schumacher, S. Schweinberger, V. Struckmeier, P. C. Trettenbrein & C. I. von Eiff. (2023). A Roadmap for Technological Innovation in Multimodal Communication Research. In: Duffy, V.G. (eds) Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. HCII 2023. Lecture Notes in Computer Science, vol 14029. 402-438. Springer, Cham. https://doi.org/10.1007/978-3-031-35748-0_30
. - Gregori, Alina & Kügler, Frank (2022). An Empirical Investigation on the Perceptual Similarity of Prosodic Language Types. In Proc. 1st International Conference on Tone and Intonation (TAI), 209-213, doi: 10.21437/TAI.2021-43 .
Prof. Dr. Patrick Georg Grosz
(Principal Investigator)
Project:
Semantics and Pragmatics of Emojis in
Digital Communication (EmDiCom)
University of Oslo
p.g.grosz@iln.uio.no
Patrick Georg Grosz is Professor of Linguistics at the University of Oslo. He obtained a Mag.phil. in Linguistics from the University of Vienna (2005) and a PhD in Linguistics from MIT (2011). His interests include semantics, syntax, pragmatics, and their interfaces; he has worked on topics such as optatives, imperatives, discourse particles, pronouns, agreement, and the diachrony of ‘dunno’ indefinites. In his current research, he is focusing on the application of linguistic methodology beyond natural language, to objects such as emojis and gestures, with a particular emphasis on face emojis.
Selected publications
- Grosz, Patrick G. (submitted): Expressivity and Emojis. Submitted for The Oxford Handbook of Expressivity edited by Daniel Gutzmann.
- Grosz, Patrick G., Gabriel Greenberg, Christian De Leon, and Elsi Kaiser (accepted): A semantics of face emoji in discourse. To appear in Linguistics and Philosophy. [https://ling.auf.net/lingbuzz/005981]
- Grosz, Patrick G. (2022): Emojis and conditionals: Exploring the super linguistic interplay of pictorial modifiers and conditional meaning. Linguistics Vanguard. [https://doi.org/10.1515/lingvan-2021-0123]
- Grosz, Patrick G., Elsi Kaiser, and Francesco Pierini (2021): Discourse anaphoricity and first-person indexicality in emoji resolution. In Proceedings of Sinn und Bedeutung 25, 340-357. [https://doi.org/10.18148/sub/2021.v25i0.941]
- Kaiser, Elsi, and Patrick G. Grosz (2021): Anaphoricity in emoji: An experimental investigation of face and non-face emoji. Proceedings of the Linguistic Society of America 6, 1009-1023. [https://doi.org/10.3765/plsa.v6i1.5067]
H – K
Dr. Alexander Henlein
(Postdoc Researcher)
Project:
Virtual Reality Sustained Multimodal Distributional Semantics for Gestures in Dialogue (GeMDiS)
Goethe University Frankfurt
henlein@em.uni-frankfurt.de
Alexander Henlein is a PostDoc at the Text Technology Lab (TTLab) of the Professorship for Computational Humanities / Text Technology of Prof. Dr. Alexander Mehler at the Goethe University Frankfurt. His doctoral research focused on the analysis of spatial semantics in language models, the extraction of object habitats from images, and the development of a VR-based Text2Scene system. Based on this work experience, he would like to develop VR-assisted communication systems within the scope of this project and use the data thus generated to create novel multimodal models.
Selected publications
- A. Henlein, A. Gopinath, N. Krishnaswamy, A. Mehler, J. Pustejovsky, “Grounding Human-Object Interaction to Affordance Behavior in Multimodal Datasets”, in Frontiers in Artificial Intelligence-Language and Computation, 2023 (accepted).
- A. Henlein and A. Mehler, “What do Toothbrushes do in the Kitchen? How Transformers Think our World is Structured,” in Proceedings of the 2022 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2022), 2022
- A. Henlein, G. Abrami, A. Kett, C. Spiekermann, and A. Mehler, “Digital Learning, Teaching and Collaboration in an Era of ubiquitous Quarantine,” in Remote Learning in Times of Pandemic – Issues, Implications and Best Practice, L. Daniela and A. Visvizin, Eds., Thames, Oxfordshire, England, UK: Routledge, 2021.
- A. Henlein, G. Abrami, A. Kett, and A. Mehler, “Transfer of ISOSpace into a 3D Environment for Annotations and Applications,” in Proceedings of the 16th Joint ACL – ISO Workshop on Interoperable Semantic Annotation, Marseille, 2020, pp. 32-35.
- A. Henlein and A. Mehler, “On the Influence of Coreference Resolution on Word Embeddings in Lexical-semantic Evaluation Tasks,” in Proceedings of The 12th Language Resources and Evaluation Conference, Marseille, France, 2020, pp. 27-33.
PD. Dr. Oliver Herbort
(Principal Investigator)
Project:
Representation of Pointing Uncertainty
for the Integration of Pointing Gestures and Speech
University of Würzburg
oliver.herbort@psychologie.uni-wuerzburg.de
Oliver Herbort is interested in human motor control, embodied choices, and the production and perception of pointing gestures. Specifically, he tries to understand the relationship between the body postures assumed during pointing, the objects or locations that are perceived to be indicated, and various situational factors. His methods include motion capture, virtual reality setups, and formal models. Oliver Herbort received his PhD from the University of Würzburg, was a post-doc in Würzburg and Tübingen, and is now an academic counsel at the University of Würzburg and PI of the VICOM project “representation of pointing uncertainty for the integration of pointing gestures and speech”.
Selected publications
- Herbort, O., Krause, L.-M., & Kunde, W. (2021). Perspective determines the production and interpretation of pointing gestures. Psychonomic Bulletin & Review, 28(2), 641-648. https://doi.org/10.3758/s13423-020-01823-7
- Herbort, O. & Kunde, W. (2016). Spatial (mis-)interpretation of pointing gestures to distal referents. Journal of Experimental Psychology: Human Perception and Performance, 42(1), 78-89. https://doi.org/10.1037/xhp0000126
- Herbort, O., & Kunde, W. (2018). How to point and to interpret pointing gestures? Instructions can reduce pointer-observer misunderstandings. Psychological Research, 82(2), 395-406. https://doi.org/10.1007/s00426-016-0824-8
- Krause, L.-M., & Herbort, O. (2021). The observer’s perspective determines which cues are used when interpreting pointing gestures. Journal of Experimental Psychology: Human Perception and Performance, 47(9), 1209-1225. https://doi.org/10.1037/xhp0000937
Prof. Dr. Annika Herrmann
(Principal Investigator)
University of Hamburg
annika.herrmann@uni-hamburg.de
Annika Herrmann is a professor for Sign Languages and Sign Language Interpreting at the Institute of German Sign Language and Communication of the Deaf (IDGS) at Universität Hamburg. Her research interests are theoretical and empirical sign language research, as well as experimental psycholinguistics. She received her PhD from University of Frankfurt am Main in 2010. After receiving her PhD, she was the Director of the Experimental Sign Language Lab at the Georg-August-University of Göttingen, has worked as a substitute professor for linguistics at the University of Cologne, and as a research assistant in an EEG-project at the University of Mainz. Annika Herrmann is co-director of the DGS-Korpus project and also co-editor of the series ‘Sign Languages and Deaf Communities’.
Selected publications
- Herrmann, A. & M. Steinbach. 2019. Expressive Gesten – expressive Bedeutungen. Expressivität in gebärdensprachlichen Erzählungen. In d’Avis, F. & R. Finkbeiner (eds.), Expressivität im Deutschen. Berlin: de Gruyter, 313–337.
- Herrmann, A. & N.-K. Pendzich. 2018. Between narrator and protagonist in fables of German Sign Language. In Hübl, A. & M. Steinbach (eds.), Linguistic Foundations of Narration in Spoken and Sign Languages. Amsterdam: Benjamins, 275–308.
- Herrmann, A. & N.-K. Pendzich. 2014. Nonmanual gestures in sign languages. In Müller, C., A. Cienki, E. Fricke, S. H. Ladewig, D. McNeill & J. Bressem (eds.), Handbook Body – Language – Communication. Berlin, Boston: De Gruyter Mouton, 2147–2160.
- Quer, J., R. Pfau & A. Herrmann. 2021. The Routledge Handbook of theoretical and experimental sign language research. London New York: Routledge, Taylor & Francis Group.
- Wienholz, A., D. Nuhbalaoglu, M. Steinbach, A. Herrmann & N. Mani. 2021. Phonological priming in German Sign Language: An eye tracking study using the visual world paradigm. Sign Language & Linguistics 24(1). 4– 35.
Prof. Dr. Stefan Hinterwimmer
(Principal Investigator)
Project:
Visual and Non-visual Means of
Perspective Taking in Language
University of Hamburg
stefan.hinterwimmer@uni-hamburg.de
Stefan Hinterwimmer’s research interests lie in the realms of semantics, pragmatics, the syntax-semantics interface, information structure and text linguistics. He has worked on adverbial quantification, free relative clauses, conditionals, topicality, specificity, pronoun resolution, the contrast between definite and demonstrative DPs and perspective taking. He received his PhD in 2006 from Humboldt-Universität zu Berlin. He is currently Akademischer Rat and Außerplanmäßiger Professor at the Department of German language and literature at the University of Wuppertal.
Selected publications
- Hinterwimmer, Stefan & Umesh Patil. 2022. The Interpretative options of anaphoric complex demonstratives. Glossa: a journal of general linguistics 7(1), 1–30.
- Hinterwimmer, Stefan, Umesh Patil & Cornelia Ebert. 2021. On the interaction of gestural and linguistic perspective taking. Frontiers in Communication 6. 625757.
- Hinterwimmer, Stefan. 2017. Two kinds of perspective taking in narrative texts. In D. Burgdorf, J. Collard, S. Maspong & B. Stefánsdóttir (eds.), Proceedings of Semantics and Linguistic Theory (SALT) 27, 282-301.
- Hinterwimmer, Stefan & David Schueler. 2015. Requantification, underquantification and partial focus in indefinites. Journal of Semantics 32, 749-797.
- Ebert, Christian, Cornelia Ebert & Stefan Hinterwimmer. 2014. A unified analysis of conditionals as topics. Linguistics and Philosophy 37(5). 353–408.
Šárka Kadavá
(PhD Candidate, PhD Contact Person)
Leibniz-Zentrum Allgemeine Sprachwissenschaft Berlin
kadava@leibniz-zas.de
Šárka Kadavá is a doctoral researcher in the DFG project On the FLExibility and Stability of gesture-speecH coordination (FLESH): Evidence from production, comprehension, and imitation. She is currently based in Leibniz-Zentrum Allgemeine Sprachwissenschaft (ZAS) in Berlin. Her research focuses on multimodality, on how different modalities contribute to the sharing of meaning and how they are coordinated with each other, and on how language-like structural features emerge in non-normative communication systems. She has also participated in a project developing Czech adaptation of MB-CDI at the Institute of Psychology, Czech Academy of Sciences.
Lisa-Marie Krause
(PhD Candidate)
Project:
Representation of Pointing Uncertainty
for the Integration of Pointing Gestures and Speech
University of Würzburg
lisa-marie.krause@uni-wuerzburg.de
Lisa-Marie started her PhD in 2019 at the University of Würzburg. In her research, she is mainly interested in the spatial interpretation of pointing gestures, especially how different factors, e.g., perspective or arm posture, are affecting this interpretation. Within ViCom, she will extend this focus by gesture-speech integration and pointing production. She aims to examine how spatial information that are expected to be communicated by pointers and the spatial information observers expected to be implied by pointing gestures match. Methodically, Lisa-Marie conducts her experiments using eye tracking in virtual reality as well as in real-life dyadic interactions.
Selected Publications
- Krause, L.-M., & Herbort, O. (2021). The observer’s perspective determines which cues are used when interpreting pointing gestures. Journal of Experimental Psychology: Human Perception and Performance, 47(9), 1209-1225. http://dx.doi.org/10.1037/xhp0000937
- Herbort, O., Krause, L.-M., & Kunde, W. (2021). Perspective determines the production and interpretation of pointing gestures. Psychonomic Bulletin & Review, 28(2), 641-648. https://doi.org/10.3758/s13423-020-01823-7
- Herbort, O. & Krause, L.-M. (2023). The efficiency of augmented pointing with and without speech in a collaborative virtual environment. In: V. G. Duffy (Ed.) Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. HCII 2023. Lecture Notes in Computer Science, vol 14028. (pp. 510-524). Springer. https://doi.org/10.1007/978-3-031-35741-1_37
- Krause, L.-M. & Herbort, O. (2023). Just visual context or part of the gesture? The role of arm orientation in bent pointing interpretation. Acta Psychologica, 241, 104062. https://doi.org/10.1016/j.actpsy.2023.104062
Dr. Anna Kuder
(Postdoc Researcher)
University of Cologne
akuder@uni-koeln.de
Anna Kuder is interested in nonmanual elements of sign languages and in the use of discourse particles across different language modalities. She obtained her PhD in 2020 at the University of Warsaw, after defending her thesis titled “Negation markers in Polish Sign Language (PJM)”. While doing her PhD she served as an Investigator or Principal Investigator in numerous projects (both research and educational) concerning PJM. She was involved in all stages of building the PJM Corpus and participated in the project of creating the first on-line Corpus-based Dictionary of PJM. As a postdoc, she is currently working at the University of Cologne, where she is conducting a comparative corpus-based study of gestural elements in Polish Sign Language, German Sign Language, and Russian Sign Language. She is also currently joining the ViCom project “Gestures or signs? Comparing manual and non-manual constructions sharing the same form in co-speech gesture and sign language: a corpus-driven approach. (GeSi)” as a postdoc.
Selected publications
- S. Gabarró López & A. Kuder (2022), A corpus-based study of the “Away gestures” across four signed languages: DGS, LSC, LSFB and PJM, Belgian Journal of Linguistics, 36, in print.
- A. Kuder (2022), Making Sign Language Corpora Comparable: A study of Palm-Up and Throw-Away in Polish Sign Language, German Sign Language, and Russian Sign Language. In: Workshop Proceedings. 10th Workshop on the Representation and Processing of Sign Languages: Multilingual Sign Language Resources. Language Resources and Evaluation Conference} (LREC), Marseille, France, 25 June 2022, ELRA, 110–117.
- A. Kuder, J. Wójcicka, P. Mostowski & P. Rutkowski (2022), Open Repository of the Polish Sign Language Corpus: Publication Project of the Polish Sign Language Corpus. In: Workshop Proceedings. 10th Workshop on the Representation and Processing of Sign Languages: Multilingual Sign Language Resources. Language Resources and Evaluation Conference (LREC), Marseille, France, 25 June 2022, ELRA, 118–123.
- A. Kuder (2021), Negation markers in Polish Sign Language (PJM), Sign Language and Linguistics, 24(1), pp. 118-131.
- A. Kuder, J. Filipczak, P. Mostowski, P. Rutkowski & T. Johnston (2018), What Corpus-based Research on Negation in Auslan and PJM Tells Us about Building and Using Sign Language Corpora. In: M. Bono et al. (eds.), Workshop Proceedings. 8th Workshop on the Representation and Processing of Sign Languages: Involving the Language Community. Language Resources and Evaluation Conference (LREC), Miyazaki, Japan, 12 May 2018, ELRA, pp. 101-106.
Prof. Dr. Frank Kügler
(Principal Investigator)
Project:
Co-speech Gestures and Prosody as
Multimodal Markers of Information Structure
Goethe University Frankfurt
kuegler@em.uni-frankfurt.de
Frank Kügler is Professor of Linguistics at Goethe University Frankfurt. His research interests are in cross-linguistic prosody from various different perspectives such as the expression and modelling of information structure, prominence and sentence mode in typologically unrelated languages, the interaction of tone and intonation, and prosodic phrasing and recursive prosodic constituents at the prosody-syntax interface. His recent interests include also the gestural marking of prosody. He has worked on the prosody of a number of typologically diverse languages including Mandarin, Hindi, Akan, Tswana, Ghanaian English, Yucatec Maya, Hungarian, and German among others.
Selected publications
- Grice, Martine & Kügler, Frank (2021) Prosodic Prominence – A Cross-Linguistic Perspective. Language and Speech, 64, 2, 253–260. https://doi.org/10.1177/00238309211015768
- Kügler, Frank (2016) Tone and intonation in Akan. In: Downing, Laura J. & Rialland, Annie (eds) Intonation in African Tone Languages. Berlin: Mouton de Gruyter, 89-129.
- Kügler, Frank, Baumann, Stefan & Röhr, Christine T. (2022) Deutsche Intonation, Modellierung und Annotation (DIMA) – Richtlinien zur prosodischen Annotation des Deutschen. In Schwarze, Cordula & Grawunder, Sven (eds.) Transkription und Annotation gesprochener Sprache und multimodaler Interaktion: Konzepte, Probleme, Lösungen. Tübingen: Narr, 23-54.
- Kügler, Frank & Sasha Calhoun. 2020. Prosodic Encoding of Information Structure: A typological perspective. In Carlos Gussenhoven & Aoju Chen (eds.), The Oxford Handbook of Language Prosody, 453–467. Oxford: OUP.
- Kügler, Frank & Caroline Féry. 2017. Postfocal downstep in German. Language and Speech 60(2). 260–288.
L – N
Dr. Silva Ladewig
(Principal Investigator)
Project:
Processes of Stabilization in Gestures.
A Media-specific and Cross-modal Approach
University of Göttingen
silva.ladewig@uni-goettingen.de
Silva Ladewig is interested in the embodied nature of language. For this she studies multimodality from a linguistic perspective in the fields of Pragmatics, Cognitive Grammar, Cognitive Semantics and Dynamic Multimodal Communication. Of special interest are stabilization processes in gestures and the interface between gesture and sign which she explores now at the Sign Language Lab at the University of Göttingen. Silva received her Ph.D. in Linguistics at the European University Viadrina in 2012 where she has worked in many interdisciplinary projects on the integration of gesture and speech, the linguistic potential of gestures, and the dynamics of multimodal metaphors.
Selected publications
- Harrison, Simon, Silva H. Ladewig & Jana Bressem (eds.) (2021). The diversity of recurrency: Recurrent gesture cross-linguistically, Gesture 20:2 (Special Issue).
- Harrison, Simon & Silva H. Ladewig (2021). Recurrent gestures throughout bodies, languages, and cultural practices. In: Gesture 20:2, 153–179.
- Ladewig, Silva H. (to appear). Recurrent gestures. In: Alan Cienki (Hg.), The Cambridge Handbook of Gesture Studies. Cambridge.
- Ladewig, Silva H. & Lena Hotze (2021). The Slapping movement as an embodied practice of dislike: Inter-affectivity in interactions among children. In: Gesture 20:2, 285–312.
- Ladewig, Silva H. (2020). Integrating Gestures. The Dimension of Multimodality in Cognitive Grammar. Berlin/ Amsterdam: De Gruyter Mouton
PD Dr. Jens Lemanski
(Principal Investigator)
Project:
Gestures and Diagrams in Visual-Spatial Communication: Methodological Tools and Applications in Mathematics and Logic
University of Münster / University of Hagen
jenslemanski@uni-muenster.de
PD Dr. Jens Lemanski is currently PI of the “History of Logic Diagrams in Kantianism” project at the University in Muenster (Thyssen-Stiftung) and Privatdozent for Philosophy at the FernUniversität in Hagen, Germany. He holds a cotutela-Ph.D in philosophy from the JGU Mainz and the Università del Salento (Lecce). He has been a research fellow at the WWU Muenster and the RU Bochum and he has published on the history and philosophy of logic, science, metaphysics, and the foundations of mathematics.
Selected publications
- J. Lemanski: World and Logic. London: College Publications, 2021.
- J. Lemanski (ed.): Language, Logic, and Mathematics in Schopenhauer. Cham: Birkhäuser, 2020.
- J. Lemanski, L. Demey: Schopenhauer’s Partition Diagrams and Logical Geometry. In A. Basu, G. Stapleton, S. Linker, C. Legg, E. Manalo & P. Viana (eds.), Diagrams 2021: Diagrammatic Representation and Inference (Lecture Notes in Artificial Intelligence 12909). pp. 149-165. 2021.
- J. Lemanski: Euler-type Diagrams and the Quantification of the Predicate, in: Journal of Philosophical Logic 49 (2): 401-416. 2020.
- J. Lemanski: Logic Diagrams, Sacred Geometry and Neural Networks, in: Logica Universalis 13 (4): 495-513. 2019.
Prof. Dr. Katja Liebal
(Principal Investigator)
Project:
Compositional structures
in chimpanzee gestural communication
University of Leipzig
katja.liebal@uni-leipzig.de
Katja Liebal obtained her PhD at the Max Planck Institute for Evolutionary Anthropology in Leipzig. She was Professor of Comparative Developmental Psychology at the Freie Universität Berlin, and is currently leading the group Human Biology and Primate Cognition at the University of Leipzig. Her main research interests include the communicative, emotional and social-cognitive abilities of non-human primates and human children from different cultural contexts, with a special focus on how developmental trajectories vary between and within humans and apes. For her research, she uses a cross-species, cross-cultural approach, combining observational and non-invasive experimental methods.
Selected publications
- Amici, F., Lembeck, M., Holodynski, M., & Liebal, K. 2022. Face to face interactions in chimpanzee and human mother-infant-dyads. Philosophical Transactions B.
- Amici, F., Oña, L., & Liebal, K. 2022. Compositionality in primate gestural communication and multicomponent signal displays. International Journal of Primatology. https://doi.org/10.1007/s10764-022-00316-9.
- Arbib, MA, Liebal, K, & Pika, S. 2008. Primate vocalization, gesture, and the evolution of human language. Current Anthropology, 49, 1053-1076.
- Liebal, K, Waller, BM, Burrows, AM, & Slocombe, KE. 2013. Primate communication: a multimodal approach. Cambridge University Press.
- Oña, L. S., Sandler, W., & Liebal, K. (2019). A stepping stone to compositionality in chimpanzee communication. PeerJ, 7, e7623. doi:10.7717/peerj.7623
Dr. Cornelia Loos
(Principal Investigator)
Project:
Exploring the Limits of Simultaneity: Encoding Caused Change-of-state Events with Classifier Constructions
in German Sign Language (DGS)
University of Hamburg
cornelia.loos@uni-hamburg.de
Cornelia Loos received her Ph.D. in Linguistics from the University of Texas at Austin in 2017. After completing a two-year postdoc project on aspects of the syntax and semantics of German Sign Language (DGS) in Göttingen as well as a second post-doc at the university of Hamburg working in the DGS-corpus project, she is currently a visiting professor at the Institute of German Sign Language and Communication of the Deaf in Hamburg. Her research interests cluster around the syntax-semantics interface, focusing on lexical semantics as well as experimental semantics and pragmatics. She works predominantly on signed languages and has investigated topics such as word class and sentencehood in American and German Sign Language (DGS), the syntax and semantics of resultative constructions in these two languages, the influence of iconicity on the semantics of taboo language in DGS, and, most recently, on response elements and NPIs in DGS.
Selected publications
- Loos, C. 2022. Sizing up adjectives: Delimiting the adjective class in American Sign Language. Sign Language & Linguistics.
- Loos, C. and D.J. Napoli. 2021. Expanding Echo: Coordinated Head Articulations as Nonmanual Enhancements in Sign Language Phonology. Cognitive Science 45(5). doi.org/10.1111/cogs.12958.
- Loos, C. 2020. Quite a mouthful: Comparing speech act verbs in Nederlandse Gebarentaal and American Sign Language. Linguistische Berichte.
- Loos, C., M. Steinbach and S. Repp. 2020. Affirming and rejecting assertions in German Sign Language. Sinn und Bedeutung 24 Proceedings.
- Loos, C., Cramer, J.M., and D.J. Napoli. 2019. Taboo terms in German Sign Language: Exploiting phonology. Cognitive Linguistics. Advance online publication. https://doi.org/10.1515/cog-2018-0077
- Loos, C., German, A., & Meier, R. P. (2022). Simultaneous structures in sign languages: Acquisition and emergence. Frontiers in Psychology, 7232.https://doi.org/10.3389/fpsyg.2022.992589
Dr. Andy Lücking
(Principal Investigator)
Project:
Virtual Reality Sustained Multimodal Distributional Semantics for Gestures in Dialogue (GeMDiS)
Goethe University Frankfurt
luecking@em.uni-frankfurt.de
Andy Lücking’s research contributes to a linguistic theory of human communication, that is, face-to-face interaction beyond single sentences. This involves the adaptation of dynamic dialogue semantics, the development of multimodal grammar extensions, occasionally the revision of traditional linguistic theories (e.g., quantification or pointing), the use of corpora and computational methods (as in the ViCom project GeMDiS), and taking an overarching cognitive perspective. Andy Lücking received a PhD in linguistics (Dr. phil.) in 2011 at Bielefeld University on iconicity and iconic gestures. He defended his habilitation in 2022 on “Aspects of multimodal communication” at the Laboratoire de Linguistique Formelle (LLF) at the Université Paris Cité.
Selected publications
- On the broader picture of multimodal communication:
Andy Lücking and Jonathan Ginzburg. “Leading voices: Dialogue semantics, cognitive science, and the polyphonic structure of multimodal interaction”. Language and Cognition , Volume 15 , Issue 1 , January 2023 , pp. 148–172, DOI: https://doi.org/10.1017/langcog.2022.30 - A dialogue- and gesture-friendly theory of quantification:
Andy Lücking and Jonathan Ginzburg. “Referential transparency as the proper treatment of quantification”. In: Semantics and Pragmatics 15, 4 (2022). doi: 10.3765/sp.15.4. (Early access: https://semprag.org/index.php/sp/article/view/sp.15.4) - A condensed, computational piece on how to assign meanings to (some kinds of) co-verbal gestures and integrate them in grammar:
Andy Lücking. “Modeling Co-verbal Gesture Perception in Type Theory with Records”. In: Proceedings of the 2016 Federated Conference on Computer Science and Information Systems. Hrsg. von Maria Ganzha, Leszek Maciaszek und Marcin Paprzycki. Vol. 8. Annals of Computer Science and Information Systems. IEEE, Sep. 2016, pp. 383–392. doi: 10.15439/2016F83. (Available at: https://annals-csis.org/proceedings/2016/drp/83.html) - Pointing gestures as search instructions:
Andy Lücking. “Witness-loaded and Witness-free Demonstratives”. In: Atypical Demonstratives. Syntax, Semantics and Pragmatics. Hrsg. von Marco Coniglio, Andrew Murphy, Eva Schlachter und Tonjes Veenstra. Linguistische Arbeiten 568. Berlin und Boston: De Gruyter, 2018, pp. 255–284. ISBN: 978-3-11-056029-9.(Preprint: https://www.researchgate.net/publication/303667514_Witness-loaded_and_Witness-free_Demonstratives)
Prof. Dr. Carina Lüke
(Principal Investigator)
Project:
Modality-Specific Effects on Language Processing
in Children with Developmental Language Disorder
University of Würzburg
carina.lueke@uni-wuerzburg.de
Carina Lüke’s research interests are in the area of communication and language acquisition in mono- and multilingual children, the identification and intervention in children with Developmental Language Disorders (DLD) and especially, the multimodal use of gestures and speech in typically an atypically developing children. She received her education in Rehabilitation Sciences (TU Dortmund University) and Clinical Linguistics (Bielefeld University), completed her Ph.D. 2015 at TU Dortmund University on the predictive value of early pointing gestures for later language skills in children. Since 2020 she is a full professor for Special Education and Therapy in Language and Communication Disorders at the University of Würzburg, were she also heads the Lab for Communication and Language (FoKuS). Within ViCom she is one of the PI of the project Modality-Specific Effects on Language Processing in Children with Developmental Language Disorder.
Selected publications
- Lüke, C., Liszkowski, U., & Ritterfeld, U. (2022). In bilinguals’ hands: Identification of bilingual, preverbal infants at risk for language delay. Frontiers in Pediatrics, 10:878163.
- Lüke, C., Ritterfeld, U., Grimminger, A., Rohlfing, K. J. & Liszkowski, U. (2020). Integrated communication system: Gesture and language acquisition in typically developing children and children with LD and DLD. Frontiers in Psychology, 11:118.
- Lüke, C., Ritterfeld, U., Grimminger, A., Liszkowski, U., & Rohlfing, K. J. (2017). Development of pointing gestures in children with typical and delayed language acquisition. Journal of Speech, Language, and Hearing Research, 60, 3185-3197.
- Lüke, C., Grimminger, A., Rohlfing, K. J., Liszkowski, U. & Ritterfeld, U. (2017). In infants’ hands: Identification of preverbal infants at risk for primary language delay. Child Development, 88, 484-492.
- Lüke, C. & Ritterfeld, U. (2014). The influence of iconic and arbitrary gestures on novel word learning in children with and without SLI. Gesture, 14, 204-225.
Prof. Dr. Alexander Mehler
(Principal Investigator)
Project:
Virtual Reality Sustained Multimodal Distributional Semantics for Gestures in Dialogue (GeMDiS)
Goethe University Frankfurt
mehler@em.uni-frankfurt.de
Alexander Mehler’s research interests include automatic analysis and synthesis of language and multimodal data in spoken and written communication. To this end, he studies multimodal and multiplex networks derived from social and communication networks using models of language evolution, machine learning, and complex network theory. He is particularly interested in measurement models that help overcome end-to-end learning and its issues. This involves models of multimodal semantics that focus on the explicit modelling of sign structures. Alexander Mehler develops and tests quantitative methods and machine learning models that merge with virtual and augmented reality. The goal of this research is to ground multimodal semantics based on human behaviour in virtual worlds.
Selected publications
- Mehler, A., Gleim, R., Gaitsch, R., Uslu, T. & Hemati, W. (2020, online). From Topic Networks to Distributed Cognitive Maps: Zipfian Topic Universes in the Area of Volunteered Geographic Information. Complexity, vol. 4, pp. 1-47. DOI: 10.1155/2020/4607025
- Mehler, A., Hemati, W., Welke, P., Konca, M. & Uslu, T. (2020, online). Multiple Texts as a Limiting Factor in Online Learning: Quantifying Dissimilarities of Knowledge Networks across Languages. Frontiers in Education (Digital Education), pp. 1-31. DOI: 10.3389/feduc.2020.562670
- Mehler A., Gleim, R., Lücking, A., Uslu, T., & Stegbauer, Ch. (2018). „On the Self-similarity of Wikipedia Talks: a Combined Discourse-analytical and Quantitative Approach“. In: Glottometrics 40, pp. 1–45
- Mehler, A., Lücking. A., & Abrami, G. (2014). „WikiNect: Image Schemata as a Basis of Gestural Writing for Kinetic Museum Wikis“. In: Universal Access in the Information Society, pp. 1–17. DOI: 10.1007/s10209-014-0386-8
- Mehler, A., Lücking, A. & Menke, P. (2012). „Assessing Cognitive Alignment in Different Types of Dialog by means of a Network Model“. In: Neural Networks 32, S. 159–164. DOI: 10.1016/j.neunet.2012.02.013
Dr. Nina-Kristin Meister
(Principal Investigator)
Project:
Parts of Speech and Iconicity in
German Sign Language (DGS)
University of Göttingen
ninakristin.meister@uni-goettingen.de
Nina-Kristin Meister (née Pendzich) received her PhD in German Linguistics from the Georg-August-University of Göttingen with a dissertation entitled “Lexical Nonmanuals in German Sign Language (DGS). An Empirical and Theoretical Investigation” (2018: Wilhelm von Humboldt-Preis German Linguistic Society (DGfS), 2017: Christian-Gottlob-Heyne-Preis Göttingen Graduate School of Humanities). Since 2017 she is the director of the Sign Language Lab at the Department of German Philology at the University of Göttingen and is involved in various research projects. Her research interests are in the field of theoretical and experimental sign language linguistics. She investigates the interfaces between gesture, emotion, and sign language, grammatical and lexical nonmanuals, narrative structures, various sentence types such as conditional sentences, and iconicity in sign languages. She has applied a variety of empirical methodologies and works with the Facial Action Coding System (FACS, Ekman et al. 2002, certified FACS-Coder). She is editing the first bimodal-bilingual book series in German Sign Language (DGS) and German ‘Deutsche Gebärdensprache und Deaf Communities’ (Buske Verlag, with Thomas Finkbeiner) and the journal ‘Linguistische Berichte’ (Buske Verlag, with Markus Steinbach).
Selected publications
- Nina-Kristin Pendzich, Jens-Michael Cramer, Thomas Finkbeiner, Annika Herrmann and Markus Steinbach (2022): How do Signers Mark Conditionals in German Sign Language? Insights from a Sentence Reproduction Task on the Use of Nonmanual and Manual Markers. In: Croatian Review of Rehabilitation Research 58 (Special Issue: Sign Language, Deaf Culture, and Bilingual Education), 206-226.
- Patrick C. Trettenbrein, Nina-Kristin Pendzich, Jens-Michael Cramer, Markus Steinbach and Emiliano Zaccarella (2021): Psycholinguistic Norms for more than 300 Lexical Signs in German Sign Language. In: Behavior Research Methods 53, 1817-1832. https://doi.org/10.3758/s13428-020-01524-y
- Nina-Kristin Pendzich (2020): Lexical Nonmanuals in German Sign Language (DGS). Empirical Studies and Theoretical Implications. Berlin: Mouton de Gruyter.
- Annika Herrmann and Nina-Kristin Pendzich (2018): Between Narrator and Protagonist in Fables of German Sign Language. In: Annika Hübl and Markus Steinbach (eds.), Linguistic Foundations of Narration in Spoken and Sign Languages. Amsterdam: Benjamins, 275-308.
- Annika Herrmann and Nina-Kristin Pendzich (2014): Nonmanual Gestures in Sign Languages. In: Cornelia Müller, Alan Cienki, Ellen Fricke, Silva H. Ladewig, David McNeill and Jana Bressem (eds.), Body – Language – Communication: An International Handbook on Multimodality in Human Interaction. Volume 2. Berlin: de Gruyter Mouton, 2149-2162.
O – R
Dr. Linda Oña
(Principal Investigator)
Project:
Compositional structures
in chimpanzee gestural communication
MPI for Human Development Berlin
linda.ona@uni-leipzig.de
Linda obtained her PhD in biology at the Max Planck Institute for Evolutionary Anthropology and Freie Universität Berlin. Currently, she holds a Post Doc position at the Max Planck Institute for Human Development in Berlin. Linda’s research interests is social cognition, with a focus on communication, in various species, including humans, dogs and non-human primates. Her research methods are based on a comparative approach using experimental and observational study designs.
Selected publications
- Oña LS, Sandler W, & Liebal K (2019). A stepping stone to compositionality in chimpanzee communication. PeerJ, 7, e7623.
- Amici F, Oña LS & Liebal K (2022). Compositionality in Primate Gestural Communication and Multicomponent Signal Displays. Int J Primatol. https://doi.org/10.1007/s10764-022-00316-9.
- Bohn M, Liebal K, Oña LS, Tessler MH (2022). Great ape communication as contextual social inference: a computational modelling perspective. Phil. Trans. R. Soc. B 377: 20210096. https://doi.org/10.1098/rstb.2021.0096.
- Liebal K, Oña LS (2018) Mind the gap – intentional gestures and emotional facial expressions of nonhuman primates? Interaction Studies, 19:1/2, pp. 121–135.
- Liebal K, Oña LS (2018) Different Approaches to Meaning in Primate Gestural and Vocal Communication. Front. Psychol. 9:478. doi: 10.3389/fpsyg.2018.00478.
Prof. Dr. Pamela Perniss
(Principal Investigator)
Project:
The Gesture-to-Sign Trajectory: Phonological Parameters
in Production and Real-Time Comprehension
University of Cologne
pperniss@uni-koeln.de
Pamela Perniss’s research takes a multimodal approach to language and has focused in particular on the role of the visual modality and iconicity in shaping language structure and processing. Following her PhD at the Max Planck Institute for Psycholinguistics (2007), she held postdoctoral positions at the Deafness, Cognition and Language (DCAL) Research Centre (University College London) and at Radboud University Nijmegen before joining the Linguistics Department at the University of Brighton. Since 2019, she is a professor in the Faculty of Human Sciences at the University of Cologne and chair of the sign language interpreting (DGS-German) program.
Selected publications
- Perniss, P., Vinson, D., & Vigliocco, G. (2020). Making sense of the hands and mouth: the role of “secondary” cues to meaning in British Sign Language and English. Cognitive Science 44(7), e12868.
- Perniss, P., Lu, J., Morgan, G., & Vigliocco, G. (2017). Mapping language to the world: The role of iconicity in the sign language input. Developmental Science, e12551, 1-23.
- Perniss, P., Özyürek, A., & Morgan, G. (2015). The influence of the visual modality on language structure and language conventionalization: Insights from sign language and gesture. Topics in Cognitive Science,7(1), 2-11.
- Perniss, P., Thompson, R.L., & Vigliocco, G. (2010). Iconicity as a General Property of Language: Evidence from Spoken and Signed Languages. Frontiers in Psychology 1, 227.
Magnus Poppe
(ViCom Data Network Manager)
Goethe University Frankfurt
m.poppe@em.uni-frankfurt.de
Dr. Wim Pouw
(ViCom Data Network Coordinator, PI)
Donders Institute &
MPI for Psycholinguistics Nijmegen
w.pouw@donders.ru.nl
(A short introduction and selected publications are to be updated.)
Prof. Dr. Pilar Prieto
(Principal Investigator)
Project:
Co-speech Gestures and Prosody as
Multimodal Markers of Information Structure
Universitat Pompeu Fabra (UPF), Barcelona
pilar.prieto@upf.edu
Pilar Prieto is an ICREA Research Professor at the Department of Translation and Language Sciences at Universitat Pompeu Fabra, Barcelona, Catalunya. Her research focuses on the communicative role of prosody and gesture in language, as well as their significance in language development and second language learning. In the last years, she has worked on the multimodal marking of Information Structure, in both adult and children’s discourse. She currently serves as associate editor of the journals Language and Speech and Frontiers in Communication, and is coediting a special issue of the “Language and Cognition” journal on Multimodal Prosody.
Selected publications
- Brown, Lucien & Pilar Prieto. 2021. Gesture and prosody in multimodal communication. In Michael Haugh, Dániel Z. Kádár & Marina Terkourafi (eds.), The Cambridge Handbook of Sociopragmatics, 430–453. Cambridge: CUP.
- Esteve-Gibert, Núria & Pilar Prieto. 2013. Prosodic structure shapes the temporal realization of intonation and manual gesture movements. Journal of Speech, Language, and Hearing Research 56(3). 850-864.
- Prieto, Pilar, C. Puglesi, J. Borràs-Comes, E. Arroyo & J. Blat. 2015. Exploring the contribution of prosody and gesture to the perception of focus using an animated agent. Journal of Phonetics 49(1). 41-54.
- Rohrer, P., Florit-Pons, M., Vilà-Giménez, I., Prieto, P. (2022). “Children use non-referential gestures in narrative speech to mark discourse elements which update common ground.” Frontiers in Psychology 12:661339. doi: 10.3389/fpsyg.2021.661339
- Vanrell, M.M., Stella, A., Gili-Fivela, B., & Prieto, P. (2013). Prosodic manifestations of the Effort Code in Catalan, Italian, and Spanish contrastive focus. Journal of the International Phonetics Association 43(2), pp.195-220.
Dr. Andrea Reichenberger
(Principal Investigator)
Project:
Gestures and Diagrams in Visual-Spatial Communication: Methodological Tools and Applications in Mathematics and Logic
University of Siegen
andrea.reichenberger@uni-siegen.de
Andrea Reichenberger is currently Research Group Leader at the Department of Mathematics, University of Siegen. Her research activities focus on women’s contributions to logic, mathematics and computer science. Previously, she worked in various research projects at German universities, among them at the University of Hagen, at the Center for the History of Women Philosophers and Scientists at Paderborn University and in the DFG research project “Thought Experiment, Metaphor, Model” at the Institute for Philosophy I at the Ruhr University Bochum. Her doctoral dissertation about Émilie du Châtelet was published by Springer in 2016.
Selected publications
- Anger, Claudia, Theodor Berwe, Alfred Olszok, Andrea Reichenberger, Jens Lemanski. 2022. “Five dogmas of logic diagrams and how to escape them.” Language & Communication 87 (1): 258-270.
- Reichenberger, Andrea. 2023. “Gender Equality and Diversity. Challenges and Perspectives for the Historiography of Science.” In: M. L. Condé and M. Salomon, eds. Handbook of the Historiography of Science. Cham: Springer (forthcoming)
- Reichenberger, Andrea. 2022. “Rózsa Péter on the Philosophy and Foundations of Mathematics: A Reappraisal.” In: J. Peijnenburg and S. Verhaegh, eds.: Women in the History of Analytic Philosophy. Cham: Springer. (forthcoming)
- Reichenberger, Andrea. 2021 “Émilie Du Châtelet on Space and Time.” In: A-L. Rey, ed.: L’épistémologie et à la philosophie des sciences d’Emilie du Châtelet. Revue d’histoire des sciences 74(2), 331–355.
- Reichenberger, Andrea. 2018 “How to Teach History of Philosophy and Science: A Digital Based Case Study.” In: R. Pisano, ed.: Methods and Cognitive Modelling in the History and Philosophy of Science–&–Education. Special Issue on HPS–&–Education in Transversal. International Journal for the Historiography of Science 5, 84–99.
S
Paula G. Sánchez Ramón
(PhD Candidate)
Project:
Co-speech Gestures and Prosody as
Multimodal Markers of Information Structure
Universitat Pompeu Fabra (UPF), Barcelona / Goethe University Frankfurt
paulaginesa.sanchez@upf.edu
Paula G. Sánchez-Ramón is a PhD student developing her research under an international joint cotutelle between Universitat Pompeu Fabra (Barcelona, Catalonia) and Goethe Universität (Frankfurt am Main, Germany). She holds a B.A. in Primary Education and Teaching (Universitat Jaume I) with a specialization in Teaching English as a Foreign Language (Universidad Internacional de La Rioja), and an M.A. in Learning Difficulties and Language Disorders with a concentration in Speech and Language (Universitat Oberta de Catalunya). As a member of the project MultIS, her research focuses on the prosodic and gestural marking of information structure in Catalan-speaking adults.
Dr. Simone Schäffner
(Principal Investigator)
Project:
Modality-Specific Effects on Language Processing
in Children with Developmental Language Disorder
University of Würzburg
simone.schaeffner@uni-wuerzburg.de
Simone Schäffner is interested in the multimodal aspects of language. Her work is focused on modality switching and the role of input-output modality compatibility while language processing. She received her PhD in Cognitive and Experimental Psychology from the RWTH Aachen University in 2018 with a dissertation entitled Modality-Specific Effects in Linguistic Multitasking. Afterwards she spent two years as a postdoc at the Cognition & Development Lab at the University of Koblenz-Landau. Since January 2021, she is working at the University of Würzburg at the department of Special Education and Therapy in Language and Communication Disorders. She is currently one of the Principal Investigators of the project Modality-Specific Effects on Language Processing in Children with Developmental Language Disorder.
Selected publications
- Schaeffner, S., Koch, I., & Philipp, A.M. (2018). Sensory-motor modality compatibility in multitasking: The influence of processing codes. Acta Psychologica, 191, 210-218. https://doi.org/10.1016/j.actpsy.2018.09.012
- Schaeffner, S., Koch, I., & Philipp, A.M. (2018). The role of learning in sensory-motor modality switching. Psychological Research, 82, 955–969. https://doi.org/10.1007/s00426-017-0872-8
- Schaeffner, S., Fibla, L., & Philipp, A.M. (2017). Bimodal language switching: New insights from signing and typing. Journal of Memory and Language, 94, 1-14. https://doi.org/10.1016/j.jml.2016.11.002
- Schaeffner, S., Koch, I., & Philipp, A.M. (2016). Semantic effects on sensory-motor modality switching. Journal of Cognitive Psychology, 28, 726-742. https://doi.org/10.1080/20445911.2016.1181636
- Schaeffner, S., Koch, I., & Philipp, A.M. (2016). The role of sensory-motor modality compatibility in language processing. Psychological Research, 80, 212-223. https://doi.org/10.1007/s00426-015-0661-1
Jun.-Prof. Dr. Tatjana Scheffler
(Principal Investigator)
Project:
Semantics and Pragmatics of Emojis in
Digital Communication (EmDiCom)
University of Bochum
tatjana.scheffler@rub.de
Tatjana Scheffler analyses communication practices in digital media, using corpus linguistic and computational methods. Current research topics include both theoretical analyses of phenomena in informal digital language (such as intensifiers, question tags, or emojis), as well as applied issues like the detection of hate speech and disinformation. She is a PI on several externally funded research projects on topics covering the variability of language in social media, the computational analysis of metaphors in online forums, disinformation detection, and the semantics and pragmatics of emojis. Tatjana received her PhD in 2008 from the University of Pennsylvania (USA), before working at the German Research Center for Artificial Intelligence (DFKI) in Berlin, as well as at the universities of Potsdam and Konstanz. In 2020, Tatjana Scheffler was appointed Assistant Professor (TT) of Digital Forensic Linguistics at Ruhr-Universität Bochum.
Selected publications
- Tatjana Scheffler, Hannah Seemann, Lesley-Ann Kern. The medium is not the message: Individual level register variation in blogs vs. tweets. Register Studies, Special Issue: Register and Social Media. 2022. https://doi.org/10.1075/rs.22009.sch
- Tatjana Scheffler, Lasse Brandt, Marie de la Fuente, and Ivan Nenchev. The processing of emoji-word substitutions: A self-paced-reading study. Computers and Human Behavior 127. 2022. doi: https://doi.org/10.1016/j.chb.2021.107076 Repository location: https://osf.io/d34y5/
- Tatjana Scheffler, Veronika Solopova, and Mihaela Popa-Wyatt. The Telegram chronicles of online harm. Journal of Open Humanities Data, 7, p.8. 2021. doi: http://doi.org/10.5334/johd.31
- Tatjana Scheffler. Conversations on Twitter. In: D. Fišer/M. Beißwenger (eds.), Investigating Computer-Mediated Communication: Corpus-Based Approaches To Language In The Digital World, Ljubljana: University Press. ISBN 978-961-237-950-6. 2017
- Scheffler, Tatjana. Two-dimensional Semantics. Clausal Adjuncts and Complements. Berlin, Boston: De Gruyter Mouton, 2013.
Dr. Martin Schulte-Rüther
(ViCom Data Network Coordinator, PI)
Project:
Multimodal Assessment of Dyadic Interaction
in Disorders of Social Interaction
Heidelberg Universtiy Hospital
martin.schulte-ruether@uni-heidelberg.de
Martin’s research is focused on social interaction and emotional processing in typical development as well as in psychiatric and neurodevelopmental disorders, in particular Autism. He employs a broad spectrum of neuroscientific and behavioral methods, including MRI, fNIRS, physiological recordings, eye-tracking, and video-based behavioral analysis. Martin studied Psychology at the Ruhr-University Bochum and received his PhD at the University of Bielefeld working in cooperation with the research Center Jülich on the cognitive neuroscience of empathy. He continued as a Post-Doc at the Child and Adolescent Psychiatry of the RWTH Aachen, focusing on neuroimaging methods in children and adolescents with autism. Next, he headed a research group for Translational Brain Research at the RWTH Aachen, Jülich-Aachen Research Alliance. Currently, he is based at the University Medical Center Göttingen, and is a senior researcher and group leader of the Social Interaction and Developmental Neuroscience Lab at the Child and Adolescent Psychiatry Göttingen.
Selected publications
- Schulte-Rüther M, Kulvicius T, Stroth S, Wolff N, Roessner V, Marschik PB, Kamp-Becker I, Poustka L (2022). Using machine learning to improve diagnostic assessment of ASD in the light of specific differential and co-occurring diagnoses. Journal of Child Psychology and Psychiatry. https://doi.org/10.1111/jcpp.13650
- Hartz A, Guth B, Jording M, Vogeley K, Schulte-Rüther M (2021). Temporal Behavioral Parameters of on-going Gaze Encounters in a Virtual Environment. Frontiers in Psychology, doi:10.3389/fpsyg.2021.673982
- Kruppa J, Reindl V, Gerloff C, Oberwelland E, Prinz J, Herpertz-Dahlmann B, Konrad K, Schulte-Rüther M. (2020). Brain-to-Brain synchrony in children and adolescents with autism spectrum disorder in parent-child-dyads. Social Cognitive and Affective Neuroscience, doi:10.1093/scan/nsaa092
- Kruppa JA, Gossen A, Großheinrich N, Oberwelland E, Cholemkery H, Freitag C, Kohls G, Fink GR, Herpertz-Dahlmann B, Konrad K, Schulte-Rüther M (2019). Social Reinforcement Learning and its Neural Modulation by Oxytocin in Autism Spectrum Disorder. Neuropsychopharmacology, 44:749-756. doi:10.1038/s41386-018-0258-7
- Burkhardt, Petra. (2006). Inferential Bridging Relations Reveal Distinct Neural Mechanisms: Evidence from Event-Related Brain Potentials. Brain and Language, 98, 2, 159-168.Oberwelland E, Schilbach L, Barisic I, Krall SC, Vogeley K, Fink GR, Herpertz-Dahlmann B, Konrad K, Schulte-Rüther M(2017). Young adolescents with autism show abnormal joint attention network: A gaze contingent fMRI study. NeuroImage Clinical, 14:112-121, doi: 10.1016/j.nicl.2017.01.006
Prof. Dr. Petra B. Schumacher
(Steering Committee Member, PI)
Project:
The Gesture-to-Sign Trajectory: Phonological Parameters
in Production and Real-Time Comprehension
University of Cologne
petra.schumacher@uni-koeln.de
Petra B. Schumacher’s research focuses on discourse processing and interface phenomena, including anaphora resolution, information structure and experimental pragmatics. In her psycho- and neurolinguistic research, she has applied a wide variety of experimental methodologies and worked with different populations. She obtained her PhD from Yale University in 2004 and held positions at the Max Planck Institutes in Leipzig and Nijmegen as well as at the University of Marburg and Mainz. She is currently professor of empirical linguistics at the University of Cologne.
Selected publications
- Patterson, Clare & Petra B. Schumacher. (2021). Interpretation Preferences in Contexts with three Antecedents: Examining the Role of Prominence in German Pronouns. Applied Psycholinguistics, 1-35.
- Röhr, Christine T., Ingmar Brilmayer, Stefan Baumann, Martine Grice & Petra B. Schumacher. (2020). Signal-Driven and Expectation-Driven Processing of Accent Types. Language, Cognition and Neuroscience, 36,1, 33-59.
- Schumacher, Petra B. (2013). When Combinatorial Processing Results in Reconceptualization: Towards a New Approach of Compositionality. Frontiers in Psychology, 4, 677.
- Schumacher, Petra B. & Yu-Chen Hung. (2012). Positional Influences on Information Packaging: Insights from Topological Fields in German. Journal of Memory and Language, 67, 2, 295-310.
- Burkhardt, Petra. (2006). Inferential Bridging Relations Reveal Distinct Neural Mechanisms: Evidence from Event-Related Brain Potentials. Brain and Language, 98, 2, 159-168.
Sarah Schwarzenberg
(PhD Candidate)
University of Hamburg
sarah.schwarzenberg@uni-hamburg.de
Sarah Schwarzenberg is a scientific researcher at the Institute of German Sign Language and Communication of the Deaf (IDGS), Universität Hamburg. Her research interests are multimodality and sign language acquisition (both L1 and L2). In her master’s thesis at Universität Hamburg, she investigated the use of finger counting and number signs and their modality-specific features. In 2021, she started her doctoral thesis on metaphors in German Sign Language (L1) and written German (L2), focusing on the identification of metaphors with a multi-methodological approach, using corpus data and experimental methods.
Selected publications
- Schwarzenberg, S. 2022. A first approach to identifying metaphors of German Sign Language in the domain of cognition. Video presented at TISLR14, 27.-30.09.2022, Osaka, Japan.
- Bauer, S. 2022. Know your terminology: On the relevance of definitions. Presentation at STaPs19, 25.- 26.03.22, München, Mainz, Köln.
- Bauer, S. 2022. Looking at place of articulation as a first approach to identifying metaphors of German Sign Language in the domain of ‘cognition’. Presentation at the DGfS44, 23.-25.02.2022, University of Tübingen.
Prof. Dr. Stefan R. Schweinberger
(Steering Committee Member, PI)
Project:
Audiovisual Perception of Emotion and Speech
in Hearing Individuals and Cochlear Implant Users
University of Jena
stefan.schweinberger@uni-jena.de
Stefan Schweinberger is interested in cognition as well as cognitive and social neuroscience. Areas of his research include person perception and human interaction, with a particular focus on communication via the face and the voice. He received his PhD from the University of Konstanz in 1991, and worked as a professor at the Universities of Glasgow (2000-2005) and Jena (2005-present). He and his team use research methods linking brain and cognition/emotion, such as event-related brain potentials (ERP), eyetracking, or investigations of patients with focal brain lesions. His research also covers individual differences and constraints to person perception or communication, whether of sensory (e.g., hearing loss) or central origin (e.g., autism, prosopagnosia).
Selected publications
- Frühholz, S., & Schweinberger, S.R. (2021). Nonverbal auditory communication – Evidence for Integrated Neural Systems for Voice Signal Production and Perception. Progress in Neurobiology, 199, 101948. doi: 10.1016/j.pneurobio.2020.101948.
- Frühholz, S., & Schweinberger, S.R. (2021). Nonverbal auditory communication – Evidence for Integrated Neural Systems for Voice Signal Production and Perception. Progress in Neurobiology, 199, 101948. doi: 10.1016/j.pneurobio.2020.101948.
- Schweinberger, S.R., & von Eiff, C.I. (2022). Enhancing Socio-emotional Communication and QoL in Young CI Recipients: Perspectives from Parameter-specific Morphing and Caricaturing. Frontiers in Neuroscience, 16:956917. doi: 10.3389/fnins.2022.956917.
- Skuk, V.G., Kirchen, L., Oberhoffner, T., Guntinas-Lichius, O., Dobel, C., & Schweinberger, S.R. (2020). Parameter-specific Morphing Reveals Contributions of Timbre and F0 Cues to the Perception of Voice Gender and Age in Cochlear Implant Users. Journal of Speech, Language, and Hearing Research, 63(9), 3155-3175. doi: 10.1044/2020_JSLHR-20-00026
- Skuk, V.G., Kirchen, L., Oberhoffner, T., Guntinas-Lichius, O., Dobel, C., & Schweinberger, S.R. (2020). Parameter-specific Morphing Reveals Contributions of Timbre and F0 Cues to the Perception of Voice Gender and Age in Cochlear Implant Users. Journal of Speech, Language, and Hearing Research, 63(9), 3155-3175. doi: 10.1044/2020_JSLHR-20-00026
Door Spruijt
(PhD Candidate)
Project:
The Gesture-to-Sign Trajectory: Phonological Parameters
in Production and Real-Time Comprehension
University of Cologne
dspruijt@uni-koeln.de
Door Spruijt’s research has focused on sign languages since her BA and MA in sign linguistics at the University of Amsterdam. She has worked extensively with both corpus and elicited data from Sign Language of the Netherlands (NGT) and the rural Balinese sign language Kata Kolok. She is now a PhD student on the Gesture-to-Sign Trajectory Project, which investigates the facilitating and inhibiting effects of iconicity and the individual phonological parameters on acquiring the lexicon in second language learners of a sign language (L2M2-learners). Understanding these effects will help improve curricula for L2M2-learners, which she, as a certified teacher of NGT, has great affinity with.
Selected publications
- Spruijt, D., P. Pamela, S. Petra B. (2023) The Contribution of Individual Parameters to Perceived Iconicity and Transparency in Gesture-Sign Pairs. FEAST. Formal and Experimental Advances in Sign Language Theory, Vol. 5, December 2023, p. 194-209, doi:10.31009/FEAST.i5.16
Prof. Dr. Markus Steinbach
(Coordinator, Steering Committee Member, PI)
Project:
Parts of Speech and Iconicity in
German Sign Language (DGS)
University of Göttingen
markus.steinbach@phil.uni-goettingen.de
Markus Steinbach is professor of Linguistics at the German Department of the Universität of Göttingen. He was educated at the University of Frankfurt/Main and obtained a PhD in Linguistics at the Humboldt-University of Berlin in 1997. In 2009, he became habilitated with a study on interface phenomena in German and German Sign Language (DGS) at the University of Mainz. His research is concerned with the influence of language modality (spoken or sign languages) on language structure, development, and processing. The main focus of his research is on the relation between form and meaning, experimental linguistics, grammaticalization, and the interaction between (sign) language and gesture. He has been a principal investigator at the Göttingen Research Center ‘Text Structures’, the Research Training Group ‘Understanding Social Relationships’ and the Research Training Group ‘Form-meaning Mismatches’. He is editing the introductory series ‘Kurze Einführungen in die Germanistische Linguistik (KEGLI)’ (Winter Verlag, with Jörg Meibauer), the sign language series ‘Sign Languages and Deaf Communities (SLDC)’ (Mouton de Gruyter and Ishara Press, with Annika Herrmann) and the journal ‘Linguistische Berichte’ (Buske Verlag, with Nina-Kristin Meister). In 2010, he established the experimental sign language linguistics research group at the Universität of Göttingen (Sign Lab Göttingen), which is running (collaborative) projects on theoretical, cognitive, historical and cultural aspects of sign languages and Deaf communities and since 2022 he has been coordinating the Priority Program ‘Visual Communication (ViCom)’ together with Cornelia Ebert.
Selected publications
- Pfau, Roland, Martin Salzmann and Markus Steinbach (2018): The Syntax of Sign Language Agreement. Common Ingredients but Unusual Recipe. Glossa: A Journal of General Linguistics 3(1), 107, 1-46.
- Steinbach, Markus (2021): Role Shift – Theoretical Perspectives. In Josep Quer, Roland Pfau and Annika Herrmann (eds.), Theoretical and Experimental Sign Language Research. London: Routledge, 351-377.
- Steinbach, Markus (2022): Differential Object Marking in Sign Languages? Restrictions on (Object) Agreement in German Sign Language. In Andrew Nevins, Anita Peti-Stantic, Mark de Vos and Jana Willer-Gold (eds.), Angles of Object Agreement. Oxford: Oxford University Press, 209-240.
- Steinbach, Markus and Edgar Onea (2016): A DRT-Analysis of Discourse Referents and Anaphora Resolution in Sign Language. Journal of Semantics 33, 409-448.
- Wienholz, Anne, Derya Nuhbalaoglu, Nivedita Mani, Annika Herrmann, Edgar Onea and Markus Steinbach (2018): Pointing to the Right Side? An ERP Study on Anaphora Resolution in German Sign Language. In: PLOS ONE 13(9), 1-19.
T – Z
Patrick Trettenbrein
(Steering Committee Member, Postdoc Researcher)
Project:
Parts of Speech and Iconicity in
German Sign Language (DGS)
MPI for Human Cognitive & Brain Sciences Leipzig
trettenbrein@cbs.mpg.de
Patrick studied cognitive sciences (linguistics, philosophy, and psychology) in Graz and London, before moving to the Max Planck Institute for Human Cognitive & Brain Sciences in Leipzig for his PhD project. His main research interest is the neurobiology of language, focusing on the modality (in-)dependence of linguistic computations and representations in the brain. In other words, in his research he doesn’t ask, “How come (only) humans can speak?”—Instead, he investigates human language as a species-specific mode of cognition independent of the modality of language use (spoken, written, or signed).
Selected publications
- Trettenbrein, P. C., Papitto, G., Friederici, A. D., & Zaccarella, E. (2021). Functional neuroanatomy of language without speech: An ALE meta‐analysis of sign language. Human Brain Mapping, 42(3), 699–712. https://doi.org/10.1002/hbm.25254
- Zaccarella, E., & Trettenbrein, P. C. (2021). Neuroscience and syntax. In N. Allott, T. Lohndal, & G. Rey (Eds.), A Companion to Chomsky (pp. 325–347). Wiley-Blackwell.
- Trettenbrein, P. C., Pendzich, N.-K., Cramer, J.-M., Steinbach, M., & Zaccarella, E. (2021). Psycholinguistic norms for more than 300 lexical signs in German Sign Language (DGS). Behavior Research Methods, 53, 1817–1832. https://doi.org/10.3758/s13428-020-01524-y
- Trettenbrein, P. C. (2016). The demise of the synapse as the locus of memory: A looming paradigm shift? Frontiers in Systems Neuroscience, 10(88). https://doi.org/10.3389/fnsys.2016.00088
Celina Isabelle von Eiff
(Postdoc Researcher)
Project:
Audiovisual Perception of Emotion and Speech
in Hearing Individuals and Cochlear Implant Users
University of Jena
celina.isabelle.von.eiff@uni-jena.de
Celina I. von Eiff is interested in cognitive and social neuroscience, particularly in voice and face perception in individuals with hearing prostheses (i.e., cochlear implants). She uses behavioural measures and EEG to investigate how the human brain integrates sensory information after cochlear implantation. She currently works at the Department of Psychology of the Friedrich Schiller University Jena.
Selected publications
- von Eiff, C. I., Kauk, J., & Schweinberger, S. R. (2023). The Jena Audiovisual Stimuli of Morphed Emotional Pseudospeech (JAVMEPS): A database for emotional auditory only, visual-only, and congruent and incongruent audiovisual voice and dynamic face stimuli with varying voice intensities. Behavior Research Methods, 56(5), 5103-5115.
- von Eiff, C. I., Frühholz, S., Korth, D., Guntinas-Lichius, O., & Schweinberger, S. R. (2022). Crossmodal Benefits to Vocal Emotion Perception in Cochlear Implant Users. iScience.
- Schweinberger, S. R. & von Eiff, C. I. (2022). Enhancing socio-emotional communication and quality of life in young cochlear implant recipients: Perspectives from parameter-specific morphing and caricaturing. Frontiers in Neuroscience, 16.
- von Eiff, C. I., Skuk, V. G., Zäske, R., Nussbaum, C., Frühholz, S., Feuer, U., Guntinas-Lichius, O., & Schweinberger, S. R. (2022). Parameter-Specific Morphing Reveals Contributions of Timbre to the Perception of Vocal Emotions in Cochlear Implant Users. Ear and Hearing, 43(4), 1178-1188.
- Nussbaum, C., von Eiff, C. I., Skuk, V. G., & Schweinberger, S. R. (2022). Vocal emotion adaptation aftereffects within and across speaker genders: Roles of timbre and fundamental frequency. Cognition, 219.
- Schweinberger, S. R., von Eiff, C. I., Kirchen, L., Oberhoffner, T., Guntinas-Lichius, O., Dobel, C., Nussbaum, C., Zäske, R., & Skuk, V. G. (2020). The Role of Stimulus Type and Social Signal for Voice Perception in Cochlear Implant Users: Response to the Letter by Meister et al. Journal of Speech, Language, and Hearing Research, 63(12), 4327-4328.
Sebastian Walter
(PhD Candidate)
Project:
Visual and Non-visual Means of
Perspective Taking in Language
Goethe University Frankfurt
s.walter@em.uni-frankfurt.de
Sebastian Walter completed his MA in linguistics at Goethe-University Frankfurt in September 2022. In October 2022 he joined the project Visual and Non-Visual Means of Perspective Taking in Language as a PhD student. His PhD thesis will investigate the interactions of perspective taking in the spoken and the gestural modality. Besides his research interests on the semantic contribution of speech-accompanying gestures and the expression of perspective in language, he is also interested in at-issueness, modals, conditionals, and verbal mood.
Selected publications
- Ebert, Cornelia, Giovanna Pirillo & Sebastian Walter. 2022. The role of gesture-speech alignment for gesture interpretation. In Sam Featherston, Robin Hörnig, Andreas Konietzko & Sophie von Wietersheim (eds.), Proceedings of Linguistic Evidence 2020, 65–77. Tübingen, Germany: University of Tübingen.
Vera Wolfrum
(PhD Candidate)
Project:
Modality-Specific Effects on Language Processing
in Children with Developmental Language Disorder
University of Würzburg
vera.wolfrum@uni-wuerzburg.de
Vera Wolfrum has completed her B.Sc. in Academic Speech Therapy/Logopedics (University of Würzburg) and M.Sc. in Teaching and Research Logopedics (RWTH Aachen University). From November 2022, she works as a PhD student within the project Modality-Specific Effects on Language Processing in Children with Developmental Language Disorder at the department of Special Education and Therapy in Language and Communication Disorders at the University of Würzburg.
Dr. Emiliano Zaccarella
(Principal Investigator)
Project:
Parts of Speech and Iconicity in
German Sign Language (DGS)
MPI for Human Cognitive & Brain Sciences Leipzig
zaccarella@cbs.mpg.de
Emiliano Zaccarella is an expert of visual, auditory and signed language processing. In his work, he uses brain imaging techniques (e.g., magnetic resonance imaging, electroencephalography, diffusion tensor imaging), behavioural methods and computational modelling to understand the general organizational principles of linguistic combinatorial abstraction in the human brain, linking ontogeny and phylogeny. He obtained his PhD in Cognitive Sciences at the University of Potsdam, in collaboration with the Max Planck Institute for Human Cognitive and Brain Sciences Leipzig and the doctoral program of the Berlin School of Mind and Brain of the Humboldt University of Berlin. He is currently group leader in the department of Neuropsychology of the Max Planck Institute for Human Cognitive and Brain Sciences Leipzig.
Selected publications
- Girard-Buttoz, C., Zaccarella, E., Bortolato, T., Friederici, A. D., Wittig R. M. & Crockford, C. (2022) Chimpanzees use numerous flexible vocal sequences with more than two vocal units: A step towards language? Communications Biology 5, 410. https://doi.org/10.1038/s42003-022-03350-8
- Trettenbrein, P. C., Papitto, G. Friederici, A. D. & Zaccarella, E. (2021) Functional neuroanatomy of language without speech, Human Brain Mapping 42 (3), 699-712. https://doi.org/10.1002/hbm.25254
- Trettenbrein, P. C., Pendzich, N. K., Cramer, J. M., Steinbach, M. & Zaccarella, E. (2021) Psycholinguistic norms for more than 300 lexical signs in German Sign Language (DGS), Behavior Research Methods 53, 1817–1832. https://doi.org/10.3758/s13428-020-01524-y
- Trettenbrein, P. C. & Zaccarella, E. (2021) Controlling Video Stimuli in Sign Language and Gesture Research: The OpenPoseR Package for Analyzing OpenPose Motion-Tracking Data in R, Frontiers in Psychology 12, 411. https://doi.org/10.3389/fpsyg.2021.628728
- Zaccarella, E., Papitto, G. & Friederici, A. D. (2020) Language and action in Broca’s area: Computational differentiation and cortical segregation, Brain and Cognition 147, 105651. https://doi.org/10.1016/j.bandc.2020.105651
Chiara Zulberti
(PhD Candidate)
Project:
Compositional structures
in chimpanzee gestural communication
University of Leipzig
chiara.zulberti@uni-leipzig.de
Chiara’s research focuses on the communicative system of one of our closest living relative: the chimpanzee. After graduating in Biology at the University of Studies of Milan, she expanded her knowledge touching topics of evolutionary anthropology, linguistics and neurosciences with her MSc in Animal Behaviour at the University of Zurich – where she collaborated with Prof. Dr. Simon Townsend and his research group. Do humans and chimpanzees share communicative abilities? What are the cognitive mechanisms enabling communication? Which selective pressures lead to the evolution of language? These are some of the broader questions involved in her research. For her PhD in the Human Biology & Primate Cognition group lead by Prof. Dr. Katja Liebal at the University of Leipzig, she is currently probing chimpanzee multimodal signals for potential combinatorial structures. Given that combinatoriality is a core feature of human language, the investigation of comparable structures in other primate species is crucial to uncover the evolutionary origins of this trait.
Mercator Fellows
Prof. Dr. Philippe Schlenker
(Mercator Fellow 2023)
CNRS (Institut Jean-Nicod, Paris)
& New York University
philippe.schlenker@gmail.com
Philippe Schlenker is a senior researcher at CNRS (Institut Jean-Nicod, Paris) and a Global Distinguished Professor at New York University. He was educated at École Normale Supérieure (Paris), and obtained a Ph.D. in Linguistics from MIT, and a Ph.D. in Philosophy from EHESS (Paris). He has taught at École Normale Supérieure, Paris, at the University of Southern California, at UCLA, and, since 2008, at NYU. P. Schlenker’s early interests included semantics, pragmatics, the philosophy of language and philosophical logic. He has conducted research on indexicals and indirect discourse, intensional semantics, anaphora, presuppositions, as well as semantic paradoxes. In recent work, he has advocated a program of ‘Super Semantics’ that seeks to expand the traditional frontiers of the field. He has investigated the semantics of sign languages, with special attention both to their logical structure and to the rich iconic means that interact with it. In order to have a point of comparison for these iconic phenomena, P. Schlenker has also investigated the logic and typology of gestures in spoken language. In collaborative work with primatologists and psycholinguists, he has laid the groundwork for a ‘primate semantics’ that seeks to apply the general methods of formal linguistics to primate vocalizations. And in recent research, he has advocated the development of a detailed semantics for music, albeit one that is very different from linguistic semantics.
Selected publications
- Signs: Schlenker and Lamberton: 2022, Meaningful Blurs: the Sources of Repetition-based Plurals in ASL. Linguistics & Philosophy 45: 201–264
- Gestures: Schlenker: 2020, Gestural Grammar. Nat. Language & Ling. Theory 38:887–936
- Signs and gestures: Schlenker; Bonnet, Marion; Lamberton, Jonathan; Lamberton, Jason; Chemla, Emmanuel; Santoro, Mirko; Geraci, Carlo: accepted with minor revisions, Iconic Syntax: Sign Language Classifier Predicates and Gesture Sequences. Linguistics & Philosophy
- Primates: Schlenker, Chemla, et al.: 2016, Formal Monkey Linguistics. Theoret. Linguistics 42,1-2:1–90
- Music: Schlenker: 2022, Musical Meaning within Super Semantics. Ling. & Philosophy 45:795–872.
Prof. Dr. Susan Goldin-Meadow
(Mercator Fellow 2024)
University of Chicago
sgsg@uchicago.edu
Susan Goldin-Meadow is the Beardsley Ruml Distinguished Service Professor in the Departments of Psychology and Comparative Human Development, and the Committee on Education, at the University of Chicago. While at Smith College during her undergraduate days, she spent her junior year at the Piagetian Institute in Geneva, which set the course of her academic career. She completed her PhD at the University of Pennsylvania under the direction of Rochel Gelman and Lila Gleitman. Her research focuses on the home-made gestures profoundly deaf children create when not exposed to sign language, and what they tell us about the fundamental properties of mind that shape language; and on the gestures hearing speakers around the globe spontaneously produce when they talk, and what they tell us about how we talk and think. She has been President of the Association for Psychological Sciences, the Cognitive Development Society, the International Society for Gesture Studies, and Chair of the Cognitive Science Society. She is a member of the American Academy of Arts and Sciences, and the National Academy of Sciences.
Selected publications
- Goldin-Meadow, S. (2023) Thinking with your hands: The surprising science behind how gestures shape our thoughts. New York, New York: Basic Books.
- Novack, M., Brentari, D., Goldin-Meadow, S., & Waxman, S. (2021). Sign language, like spoken language, promotes object categorization in young hearing infants. Cognition, (Vol. 215). Elsevier BV.
- Wakefield, E., & Goldin-Meadow, S. (2021). How gesture helps learning: Exploring the benefits of gesture within an embodied framework. In S. Stolz (ed.), The body, embodiment, and education: An interdisciplinary approach. Routledge.
- Goldin-Meadow, S. The resilience of language: What gesture creation in deaf children can tell us about how all children learn language. N.Y.: Psychology Press (a subsidiary of Taylor & Francis), 2003.
- Goldin-Meadow, S. Hearing gesture: How our hands help us think. Cambridge, MA.: Harvard University Press, 2003.
Prof. Dr. Sotaro Kita
(Mercator Fellow 2025)
University of Warwick
S.Kita@warwick.ac.uk
Sotaro Kita is Professor of Psychology of Language at the University of Warwick, in the UK. After a bachelor’s degree in mathematical engineering and a master’s degree in information engineering from University of Tokyo. He obtained a PhD in linguistics and psychology from the University of Chicago. Immediately after his PhD, he laid a foundation for gesture research in Nijmegen; he established and led the “Gesture Project” at Max Planck Institute for Psycholinguistics, in the Netherlands, from 1993 to 2003. Then, he held a faculty position at University of Bristol and University of Birmingham, before joining University of Warwick. He served as the President of the International Society of Gesture Studies and also as the Editor of the journal, GESTURE. His research primarily concerns the interplay among gesture, language and cognition.
He played a key role in establishing research on gesture as an important part of psycholinguistics. He has investigated how gesture relates to various aspects of language: motion events (Kita & Özyürek, 2003, Journal of Memory and Language), metaphor (Argyriou, Mohr & Kita, 2017, JEP: Learning, Memory and Cognition), and discourse contexts (Friz, Kita, Littlemore, & Krott, 2021, Journal of Memory and Language). He has also investigated how gesture varies cross-culturally (Kita, 2009, Language and Cognitive Processes), and how gesture production shapes gesturer’s conceptualization for speaking and thinking (Kita, Alibali, & Chu, 2017, Psychological Review).
He has also investigated the role of gesture in language development and spatial cognition. For example, he showed that deaf Nicaraguan children turned gesture into an emerging sign language (Senghas, Kita & Özyürek, 2004, Science) and that children learn a novel word better when they see an adult’s gesture depicting the word referent (Aussems & Kita, 2021, Child Development). He also showed that gesture production facilitates spatial cognition (Chu & Kita, 2011, JEP: General).
Selected publications
- Aussems, S., & Kita, S. (2021). Seeing iconic gesture promotes first- and second-order verb generalization in preschoolers. Child Development, 92(1), 124-141. https://dx.doi/org/10.1111/cdev.13392
- Chu, M., & Kita, S. (2011). The nature of gestures’ beneficial role in spatial problem solving. Journal of Experimental Psychology: General, 140(1), 102-115. http://dx.doi.org/10.1037/a0021790
- Fritz, I., Kita, S., Littlemore, J., & Krott, A. (2021). Multimodal language processing: How preceding discourse constrains gesture interpretation and affects gesture integration when gestures do not synchronise with semantic affiliates. Journal of Memory and Language, 117, 104191. doi:https://doi.org/10.1016/j.jml.2020.104191
- Kita, S., Alibali, M. W., & Chu, M. (2017). How do gestures influence thinking and speaking? The gesture-for-conceptualization hypothesis. Psychological Review, 124(3), 245-266. http://dx.doi.org/10.1037/rev0000059
- Kita, S., Alibali, M. W., & Chu, M. (2017). How do gestures influence thinking and speaking? The gesture-for-conceptualization hypothesis. Psychological Review, 124(3), 245-266. http://dx.doi.org/10.1037/rev0000059
- Kita, S., & Emmorey, K. (2023). Gesture links language and cognition for spoken and signed languages. Nature Reviews Psychology, 2, 407-420.
- Kita, S., & Özyürek, A. (2003). What does cross-linguistic variation in semantic coordination of speech and gesture reveal?: Evidence for an interface representation of spatial thinking and speaking. Journal of Memory and Language, 48, 16-32. http://dx.doi.org/10.1016/S0749-596X(02)00505-3
Consultants
Dr. Liona Paulus
(Accessibility Consultant)
University of Hamburg
liona.paulus@uni-hamburg.de
Dr. Liona Paulus grew up bilingually with German Sign Language (DGS) and German. She obtained her PhD in Linguistics at the University of Göttingen with a sociolinguistic study on conditional clauses in DGS and Brazilian Sign Language (Libras). She is a state certified interpreter for DGS and German and currently teaches Deaf Studies, Translation Studies and Interpreting Practice as a research assistant at the University of Cologne. In addition, she provides trainings on linguistics for sign language lecturers and teachers and is part of the examination committee of the teachers’ academy in Darmstadt. Her research interests comprise e.g., the development of terminology in sign languages in the field of sciences, technology, engineering and math (STEM), the use of gestures by sign language interpreters and comparative sign language research. In the priority program ViCom, her task is to create an interface between the projects and the Deaf communities and to advise stakeholders on issues regarding accessibility and diversity.
Selected publications
- Thomas Finkbeiner, Liona Paulus and Nina-Kristin Meister (in prep.): 100 Fragen und Antworten rund um die Deutsche Gebärdensprache (DGS). Hamburg: Buske.
- Paulus, L. (2021). Der Konditionalsatz in Deutscher Gebärdensprache (DGS) und Brasilianischer Gebärdensprache (Libras) – Eine empirische soziolinguistische Studie. Veröffentlichte Doktorarbeit. Georg-August-Universität Göttingen. Verfügbar unter: http://hdl.handle.net/21.11130/00-1735-0000-0008-58D1-9.
- Paulus, L. (2020). Chapter 2. Socio-Historical Background. The sign language community. In: Proske, S., D. Nuhbalaoglu, A. Herrmann, J. Hosemann & M. Steinbach (eds.), A Grammar of German Sign Language (DGS). 25–30. SIGN-HUB Sign Language Grammars, doi: http://doi.org/10.48406/texn-2f63.
- Riegert, K., Döring, C.; Friedrich-Austermühle, S.; Pallaske, J.; Paulus, L. & Temming, T. (2019). Gebärdensprachdolmetschen. In: Gätjens, J., B. Luz & S. Osterberg (Hgg.). Handbuch Dolmetschen. Grundlagen und Praxis. 113–137. Berlin: BDÜ Fachverlag.
Dr. Marc Schulder
(Open Data Consultant)
University of Hamburg
marc.schulder@uni-hamburg.de
Marc is a computational linguist working to further the open and ethical creation and use of language data, especially data for signed languages. After completing his PhD on the automatic creation of lexical resources for negation-causing content words in English and German, he joined the DGS-Korpus project, creators of the largest available discourse corpus of German Sign Language (DGS), in early 2019. As part of the project he investigates machine-assisted methods to support corpus linguistic research, possible implementations of open science principles and data standardisation for signed language data, and the creation of cross-lingual sign language resources. Marc is also co-creator of the Sign Language Dataset Compendium, a website collecting information on corpora and lexical resources for sign languages from around the globe, and of the workshop series archive sign-lang@LREC Anthology.
Selected publications
- Marc Schulder and Thomas Hanke. 2022. How to be FAIR when you CARE: The DGS Corpus as a Case Study of Open Science Resources for Minority Languages. In Proceedings of the 13th International Conference on Language Resources and Evaluation (LREC 2022), pages 164–173, Marseille, France. European Language Resources Association. URL: https://www.sign-lang.uni-hamburg.de/lrec/pub/2022.lrec-1.18.html
- Maria Kopf, Marc Schulder, and Thomas Hanke. 2022. The Sign Language Dataset Compendium: Creating an Overview of Digital Linguistic Resources. In Proceedings of the 10th Workshop on the Representation and Processing of Sign Languages(sign-lang@LREC 2022), pages 102–109, Marseille, France. European Language Resources Association. URL: https://www.sign-lang.uni-hamburg.de/lrec/pub/22025.html
- Gabriele Langer and Marc Schulder. 2020. Collocations in Sign Language Lexicography: Towards Semantic Abstractions for Word Sense Discrimination. In Proceedings of the 9th Workshop on the Representation and Processing of Sign Languages (sign-lang@LREC 2020), pages 127–134, Marseille, France. European Language Resources Association. URL: https://www.sign-lang.uni-hamburg.de/lrec/pub/20017.html
- Thomas Hanke, Marc Schulder, Reiner Konrad, and Elena Jahn. 2020. Extending the Public DGS Corpus in Size and Depth. In Proceedings of the 9th Workshop on the Representation and Processing of Sign Languages(sign-lang@LREC 2020), pages 75–82, Marseille, France. European Language Resources Association. URL: https://www.sign-lang.uni-hamburg.de/lrec/pub/20016.html
- Marc Schulder, Michael Wiegand, and Josef Ruppenhofer. 2021. Automatic generation of lexica for sentiment polarity shifters. Natural Language Engineering, 27(2):153–179. URL: https://doi.org/10.1017/S135132492000039X
PhD Contact Persons
Alina Gregori
(Steering Committee Member, PhD Candidate, PhD Contact Person)
Project:
Co-speech Gestures and Prosody as
Multimodal Markers of Information Structure
Goethe University Frankfurt
gregori@lingua.uni-frankfurt.de
Alina Gregori received her Bachelors (2021) and Masters (2022) degree in theoretical linguistics at Goethe University Frankfurt with a focus on Phonology. She started working as a PhD student in the project MultIS in October 2022. Within the project, she investigates the prosody-gesture link in communication and the impact of information structure (focus, topic, givenness) on the synchronization of gestures and prosodic entities in German. The bigger picture of the project includes a comparative analysis of German and Catalan with regard to the multimodal marking of information structure. A central aim in MultIS is the empirical approach to previously established prosody-gesture models, considering experimental as well as spontaneous speech utterances. Alina Gregoris MA thesis (title: “Co-speech Gestures, Information Structure and Prosody: A Corpus Study on Prominence Peak Alignment”) served as a pilot and preparation for MultIS.
Selected publications
- Gregori, Alina & Kügler, Frank (2022). An Empirical Investigation on the Perceptual Similarity of Prosodic Language Types. In Proc. 1st International Conference on Tone and Intonation (TAI), 209-213, doi: 10.21437/TAI.2021-43.
Šárka Kadavá
(PhD Candidate, PhD Contact Person)
Leibniz-Zentrum Allgemeine Sprachwissenschaft Berlin
kadava@leibniz-zas.de
Šárka Kadavá is a doctoral researcher in the DFG project On the FLExibility and Stability of gesture-speecH coordination (FLESH): Evidence from production, comprehension, and imitation. She is currently based in Leibniz-Zentrum Allgemeine Sprachwissenschaft (ZAS) in Berlin. Her research focuses on multimodality, on how different modalities contribute to the sharing of meaning and how they are coordinated with each other, and on how language-like structural features emerge in non-normative communication systems. She has also participated in a project developing Czech adaptation of MB-CDI at the Institute of Psychology, Czech Academy of Sciences.
Vera Wolfrum
(PhD Candidate)
Project:
Modality-Specific Effects on Language Processing
in Children with Developmental Language Disorder
University of Würzburg
vera.wolfrum@uni-wuerzburg.de
Vera Wolfrum has completed her B.Sc. in Academic Speech Therapy/Logopedics (University of Würzburg) and M.Sc. in Teaching and Research Logopedics (RWTH Aachen University). From November 2022, she works as a PhD student within the project Modality-Specific Effects on Language Processing in Children with Developmental Language Disorder at the department of Special Education and Therapy in Language and Communication Disorders at the University of Würzburg.
Associate Members
Carly A. Anderson
(Associate Member)
University College London
carly.anderson@ucl.ac.uk
Carly Anderson is a UCL Senior Research Fellow and a Principal Investigator based at both UCL’s Institute of Cognitive Neuroscience and at the Deafness Cognition and Language Research Centre. She is currently funded by a Sir Henry Wellcome Postdoctoral Research Fellowship and hosted by Professor Mairéad MacSweeney in the Visual Communication Research Group. She is a Visiting Researcher on Professor Stefan Schweinberger et al.’s ViCom project.
As a hearing child of congenitally deaf grandparents, and the daughter of a British Sign Language (BSL) Interpreter, BSL became her second language. She conducts her research through a diverse and inclusive lens informed by her experiences and the expert lived experiences of the Deaf community.
During her PhD she examined how the brain adapts after cochlear implantation in deaf adults. Importantly, she studied the neglected role of visual information and how this is combined with auditory information during social communication. Her current research focusses on how the brain coordinates visual information from the face and auditory information from the voice to support language and emotion perception. She is interested in how deafness and different language experiences, including accessing speech via a cochlear implant and bilingualism, can shape these processes. Her research has implications for optimising real-world language outcomes for people who are deaf and use a cochlear implant by taking a multimodal perspective of human communication and the brain.
Selected publications
- Anderson, C.A., Cushing, S.L., Papsin, B.C., Gordon, K.A. (2022). Cortical imbalance following delayed restoration of bilateral hearing in deaf adolescents. Human Brain Mapping, 43 (12), 3662-3679. https://doi.org/10.1002/hbm.25875
- Anderson, C.A., Wiggins, I.M., Kitterick, P.T., Hartley, D.E.H. (2017). Adaptive benefit of cross-modal plasticity following cochlear implantation in deaf adults. Proceedings of the National Academy of Sciences of the United States of America (PNAS), 114 (38), 10256-10261. https://doi:10.1073/pnas.1704785114
- Anderson, C.A., Lazard, D.S., Hartley, D.E.H. (2017). Plasticity in bilateral superior temporal cortex: Effects of deafness and cochlear implantation on auditory and visual speech processing. Hearing Research, 343, 138-149. https://doi:10.1016/j.heares.2016.07.013
Conference abstracts: - Gregoire-Mitha, N., Barton, J. J., MacSweeney, M., & Anderson, C. A. (2022). Impact of visual speech on gaze following in monolingual and bilingual adults. Journal of Vision, 22(14), 3360-3360. https://doi.org/10.1167/jov.22.14.3360
- Anderson, C., Schweinberger, S. (2022). Multimodal socio-emotional communication: basic mechanisms and functioning in altered sensory and central conditions. Psychophysiology. https://doi.org/10.1111/psyp.14139
Camila Antonio Barros
(Associate Member, PhD Candidate)
Free University of Berlin
c.antonio.barros@fu-berlin.de
Camila Antonio Barros is currently working on a PhD project that explores the coordination between gesture and speech in Spanish and Portuguese, focusing on how gestures align with phonological, semantic, and pragmatic domains. In the initial phase, a corpus of recipe descriptions from Spanish (Bogotá, Madrid) and Portuguese (Lisbon, São Paulo) speakers was annotated to examine how prosodic domains are associated with phonological anchors, semantic meanings, and pragmatic functions. Preliminary findings suggest that gestures are linked to speech through complex, context-dependent associations, often diverging from McNeillian synchronicity rules. The next phase will use experimental paradigms to further explore the differences between synchronicity and alignment.
Selected publications
- Barros, C. A., Reich, U., Dufter, A. Revisiting Rhythm in Romance Languages. In press. In Lars Meyer & Antje Strauss (eds.), Rhythms of Speech and Language: Culture, Cognition, and the Brain. Cambridge University Press.
- Barros, C. A., Ciprián-Sánchez, J. F., & Santos, S. M. (2024). A Tool for Determining Distances and Overlaps between Multimodal Annotations. In N. Calzolari, M.-Y. Kan, V. Hoste, A. Lenci, S. Sakti, & N. Xue (Eds.), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) (pp. 1705–1714). ELRA and ICCL.
- arros, C. A. & Mello, H. (2023). The C-ORAL-BRASIL proposal for the treatment of multimodal corpora data: the BGEST corpus pilot project. In: Ramírez, A. G.; Mejía, J. M.; Martin, P. V. Digital Humanities, Corpus and Language Technology / Humanidades Digitales, Corpus y Tecnología del Lenguaje. University of Groningen Press. DOI: 10.21827/64db66b6f1039
- Barros, C. A. & Mello, H. (2023). Metodologia de alinhamento som/gesto: o que nos revela a fala espontânea. In: da Hora, D. Helmer, A. Interseções Linguísticas: Estudos diversos. São Paulo: Líquido Editorial. p. 54-70.
Kathryn Barnes
(Associate Member)
Goethe University Frankfurt
barnes@lingua.uni-frankfurt.de
Kathryn Barnes is a doctoral researcher in formal semantics at Goethe University. She completed her BA in French and German Studies at the University of Warwick in 2016 and her MA in Linguistics at The University of Manchester in 2019, where her thesis was based on fieldwork on the semantics of modal verbs in Malay. Her research has since turned to iconicity, with a particular focus on ideophones, including experimentally investigating the meaning and at-issue status of ideophones crosslinguistically, as well as developing a semantic account for the phenomena. She is also interested in exploring how ideophones interact with other iconic enrichments such as gestures and understanding ideophone-like constructions in other modalities, for example classifier constructions and idiomatic signs in sign language.
Selected publications
- Barnes, K. R., Ebert, C., Hörnig, R., & Stender, T. (2022). The at-issue status of ideophones in German: An experimental approach. Glossa: A Journal of General Linguistics, 7(1). https://doi.org/10.16995/glossa.5827
Marion Bonnet
(Associate Member, PhD Candidate)
University of Göttingen
marion.bonnet@uni-goettingen.de
Marion Bonnet graduated in 2021 from the University of Paris, France, where she received her Bachelor and Master in Theoretical and Experimental Linguistics. Since September 2021, she is affiliated with Göttingen University and works as a PhD student for the IDEAlISM project (collaboration of UCL, Frankfurt and Göttingen University). Her research focuses mainly on pointing gestures and their interaction with speech at the semantic-pragmatic interface, aiming at a more complete understanding of multimodal communication and a potential enrichment of linguistic models. She proposes to investigate this topic by adopting an experimental approach fed by sign language, semantic and pragmatic theories.
Selected publications
- Bonnet, M., Donati, C., & Geraci, C. (2022). Evidence for early lexical integration of speech and gestures. In NELS 52: Proceedings of the Fifty-Second Annual Meeting of the North East Linguistic Society (Vol. 1, pp. 81-94).
- Schlenker, P., Bonnet, M., Lamberton, J., Lamberton, J., Chemla, E., Santoro, M., & Geraci, C. (2022). Iconic Syntax: Sign Language Classifier Predicates and Gesture Sequences. Linguistics and Philosophy.
Jana Bressem
(Associate Member)
TU Chemnitz
jana.bressem@phil.tu-chemnitz.de
Jana Bressem is Post-Doc at the chair for German Linguistics, Semiotics and Multimodal Communication at the TU Chemnitz. She is the head of the center „Gesture Studies and Speech Sciences“, executive board member and project leader in the CRC 1410 “Hybrid Societies” in a project on “Intentionality and joint attention in multimodal interaction”. Jana Bressem received her PhD at the European University Viadrina in 2012 where she has also worked in various (interdisciplinary) research projects. She is interested in the multimodality of language (speech/gesture, text/image), typological gesture studies, language and cognition, pragmatics, and human-machine interaction.
Selected publications
- Bressem, J. (2015). Repetition als Mittel der Musterbildung bei redebegleitenden Gesten. In C. Dürscheid & J.G. Schneider (eds.), Satz, Äußerung, Schema (pp. 422-442). Berlin, Boston: de Gruyter Mouton.
- Bressem, Jana (2021). Repetitions in gesture: A cognitive-linguistic and usage based perspective. Berlin, Boston: De Gruyter: Mouton. Bressem, J., & Ladewig, S. H. (2011). Rethinking gesture phases: Articulatory features of gestural movement? Semiotica, 184(1/4), 53–91. doi:10.1515/semi.2011.022
- Bressem, J., Ladewig, S. H., & Müller, C. (2013). Linguistic annotation system for gestures (LASG). In C. Müller, A. Cienki, E. Fricke, S. H. Ladewig, D. McNeill, & S. Teßendorf (eds.), Body-Language-Communication: An International Handbook on Multimodality in Human Interaction. (Handbooks of Linguistics and Communication Science 38.1.) (pp. 1098-1125). Berlin, Boston: De Gruyter Mouton.
- Bressem, J., Stein, N., & Wegener, C. (2017). Multimodal language use in Savosavo: Refusing, excluding and negating with speech and gesture. Pragmatics, 27(2),173-206. doi:10.1075/prag.27.2.01bre
- Harrison, Simon, Silva H. Ladewig & Jana Bressem (eds.) (2021). The diversity of recurrency: Recurrent gesture cross-linguistically, Gesture 20:2 (Special Issue).
Ingmar Brilmayer
(Associate Member)
University of Cologne
ibrilmay@uni-koeln.de
Ingmar Brilmayer is a Postdoctoral Researcher in the research project “Communication Electrified” (Volkswagenstiftung) with Prof. Petra Schumacher at the University of Cologne. Since 2020 he works on language in interaction and the integration of multimodal cues using combined EEG and eye tracking recordings with freely moving participants. Ingmar Brilmayer received his Bacherlor’s degree in Linguistics at the University of Leipzig (Leipzig, Germany), and his Master’s degree in Psycholinguistics, as well as his PhD at the University of Mainz (Mainz, Germany).
Selected publications
- Brilmayer, I., Werner, A., Primus, B., Bornkessel-Schlesewsky, I., & Schlesewsky, M. (2019). The exceptional nature of the first person in natural story processing and the transfer of egocentricity. Language, Cognition and Neuroscience, 34(4), 411-427.
- Röhr, C. T., Brilmayer, I., Baumann, S., Grice, M., & Schumacher, P. B. (2021). Signal-driven and expectation-driven processing of accent types. Language, Cognition and Neuroscience, 36(1), 33-59.
- Brilmayer, I., & Schumacher, P. B. (2021). Referential chains reveal predictive processes and form-to-function mapping: An electroencephalographic study using naturalistic story stimuli. Frontiers in Psychology, 12, 623648
Valentina Colasanti
(Associate Member)
Trinity College Dublin
Valentina.Colasanti@tcd.ie
Valentina Colasanti is an Assistant Professor in Linguistics at Trinity College Dublin. Her research profile is situated within both theoretical linguistics and Romance linguistics (especially the Italo-Romance sub-family of languages). Most of her publications involve applying generative linguistic theory to syntactic and semantic phenomena from Italo-Romance.
In 2019, her interests broadened into the study of phenomena expressed within the visual modality, i.e. gesture. Such phenomena have been almost entirely overlooked in theoretical syntax and in Romance linguistics more generally. Her work in this area adopts a purely formal approach, drawing empirically from the gesture-rich languages of Southern Italy in particular. In 2020, she established the Gestural Grammar Lab (GestuGram Lab) at Trinity College Dublin after successfully competing for funding streams. She is currently directing the lab, which so far comprises two PhD students, one lab manager, five research assistants (postgraduate and undergraduate), and two fieldworkers.
Selected publications
- Colasanti, Valentina (2023). Gestural focus marking in Italo-Romance, Isogloss. Open Journal of Romance Linguistics, 9/(4)/5, 1-39. doi: https://doi.org/10.5565/rev/isogloss.296.
- Colasanti, Valentina (2023). Functional gestures as morphemes: Some evidence from the languages of Southern Italy, Glossa: a journal of general linguistics 8(1), 1–45. doi: https://doi.org/10.16995/glossa.9743.
- Colasanti, Valentina (2023). Auxiliary selection in Southern Lazio. Some implications for Romance microvariation and its limits, L’Italia dialettale, LXXXIV, 97–150.
- Colasanti, Valentina (2022). Micro-contact in Southern Italy: language change in Southern Lazio under pressure from Italian, Languages 2022, 7, 286. doi: https://doi.org/10.3390/languages7040286.
- Colasanti, Valentina (2021). Criterial positions as diagnostics in Italo-Romance: some highs and lows. Revue roumaine de linguistique, LXVI, 2-3, 265–282.
- Colasanti, Valentina and Martina Wiltschko (2019). Spatial and discourse deixis and the speech act structure of nominals, Proceedings of the Canadian Linguistic Association Annual Meeting (CLA), 1-3 June 2019, Vancouver BC.
Arianna Colombani
(Associate Member, PhD Candidate)
University of Potsdam
colombani@uni-potsdam.de
I am a psycholinguist focusing on gesture learning and multimodal communication in hearing and hearing-impaired populations. I came to academia after several years of working as an Italian Sign Language interpreter and a school-based educator. I am currently a PhD student in Linguistics and Cognitive Sciences at the University of Potsdam (Germany) and Macquarie University (Australia) within the IDEALAB PhD program, under the joint supervision of Prof. Outi Tuomainen, Prof. Mridula Sharma, Prof. Natalie Boll-Avetisyan, and Dr. Amanda Saksida.
My PhD research investigates whether statistical learning mechanisms, well-established in spoken word learning, also apply to language expressed in the visual modality, with a specific focus on symbolic gestures. To address this question, I study gesture learning across development using EEG and eye-tracking measures.
Selected publications
- Saksida A., Rebesco R., Scuderi M., Colombani A., Messineo I., Pavani F., Orzan E., Pintonello S., (2025). The role of symbolic gestures in the path towards auditory rehabilitation of infants with hearing loss: a feasibility study. Deafness & Education International, https://doi.org/10.1080/14643154.2025.2541433
- Colombani, A., Peter V., Quian, Y.M., Saksida A., Boll-Avetisyan N., Tuomainen O., Sharma, M. (2025), Cross-situational learning of sign-like gestures in children and adults: a behavioural and event-related potential study. Language, Cognition and Neuroscience https://doi.org/10.1080/23273798.2025.2539136
- Rebesco, R., Colombani, A., Handjaras, G., Bottari, D. & Orzan, E. (2024). Early assessment of communicative competence in children with hearing loss using the Child-Caregiver Communication Assessment through Rebesco’s Evaluation (CC-CARE) method. International Journal of Pediatric Otorhinolaryngology. 111927. https://10.1016/j.ijporl.2024.111927
- Saksida, A., Rebesco, R., Colombani, A., Pintonello, S., Tonon, E., Santoro, AM. and Orzan, E. (2024) The timeline of non-vocal and vocal communicative skills in infants with hearing loss. Front. Pediatr.11:1209754. https://doi:10.3389/fped.2023.1209754
- Colombani, A., Saksida, A., Pavani, F., & Orzan, E. (2022). Symbolic and deictic gestures as a tool to promote parent-child communication in the context of hearing loss: a systematic review. International Journal of Pediatric Otorhinolaryngology,165 https://doi.org/10.1016/j.ijporl.2022.111421
Pia Gehlbach
(Associate Member)
University of Göttingen
pia.gehlbach@uni-goettingen.de
Pia Gehlbach works on iconicity and semantic conceptualization in sign language with a focus on German Sign Language. She holds a B.A. in English Philology and General Linguistics and a M.A. in English with a linguistic focus, both from the Georg-August-University of Göttingen where she now works as a member of the Sign Language Lab at the Department of German Philology. She is a member of the RTG 2070 Understanding Social Relationships. In her PhD project, she investigates the influence of sign language iconicity on the semantic conceptualization of various types of concepts, combining corpus-data analysis with an experimental approach. Her project aims at providing a systematic overview of the iconicity of the investigated signs, as well as to examine if, and how, this iconicity has an impact on the way in which a concept is semantically conceptualized.
Jonas Hartke
(Associate Member)
University of Göttingen
jhartke@posteo.de
Jonas Hartke holds a B.A. in German Philology and Protestant Religion and a M.A. in German Philology, both from the Georg-August-University of Göttingen. His main research interests are the semantics-pragmatics interface, gestures and pragmatic phenomena like irony and lies. At the moment he studies German Philology and Protestant Religion in the Master of Education.
Katja Jasinskaja
(Associate Member)
University of Cologne
katja.jasinskaja@uni-koeln.de
Iconicity, i.e. similarity between form and content, is present in visual communication to greater extent than it is in spoken and written language, and recent theoretical work on iconicity is largely based on the study of sign languages, gesture, and pictorial language. I am currently developing a research agenda whose goal is to study temporal iconicity in narrative discourse – the way the order and duration of events in the story are represented by the order and duration of utterances that report those events. The iconic representation of the order of events is probably the most standard example of iconicity in non-visual communication, and the extension of the idea to other temporal characteristics of events is the main new contribution of the envisaged research.
Selected publications
Jasinskaja, K. (2024). Discourse Time is Real. Talk presented at the Workshop Incremental constructions within and across languages at the 35th European Summer School in Logic, Language, and Information, ESSLLI 2024. https://lingbuzz.net/lingbuzz/008461. Leuven, Belgium.
Jasinskaja, K. and F. Zickenheiner (2024). “Speech acts that support other speech acts”. https://lingbuzz.net/lingbuzz/008455.
Jasinskaja, K. and E. Karagjosova (2020). “Rhetorical relations”. In: The Companion to Semantics. Ed. by D. Gutzmann, L. Matthewson, C. Meier, H. Rullmann, and T. E. Zimmermann. Oxford: Wiley.
Jasinskaja, K. (2013a). “Corrective Elaboration”. In: Lingua 132, pp. 51–66.
Jasinskaja, K. (2013b). “Iconic update”. In: Znaki czy nie znaki. Ed. by M. Guławska-Gawkowska and G. Zeldowicz. Warszawa: Wydawnictwa Uniwersytetu Warszawskiego, pp. 87–114.
Jasinskaja, K. and A. Rossdeutscher (2012). “Through Narrative Planning Towards the Preverbal Message: A DRT-based Approach”. In: Constraints in Discourse 3: Representing and inferring discourse structure. Ed. by A. Benz, P. Kühnlein, and M. Stede. John Benjamins, pp. 45–76.
Jasinskaja, K. (2010). “Modelling Discourse Relations by Topics and Implicatures: The Elaboration Default”. In: Constraints in Discourse 2. Ed. by A. Benz, P. Kühnlein, and C. Sidner. Amsterdam: John Benjamins, pp. 61–79.
Jasinskaja, K. and A. Rossdeutscher (2008). “Towards Generating Narratives from a Preverbal Message: A DRT-based approach”. In: Constraints in Discourse 3. Proceedings of the Workshop. Ed. by A. Benz, P. Kühnlein, and M. Stede. University of Potsdam. Potsdam, Germany, pp. 45–52.
Zeevat, H. and K. Jasinskaja (2007). “And as an additive particle”. In: Language, Representation and Reasoning. Memorial volume to Isabel Gómez Txurruka. Ed. by M. Aurnague, K. Korta, and J. M. Larrazabal. University of the Basque Country Press, pp. 315–340.
Marianthi Koraka
(Associate Member)
University of Göttingen
marianthi.koraka@uni-goettingen.de
Marianthi works on sign languages and holds a special interest in the aspect of modality effects. She completed her BA and MA studies at the University of Ioannina, Greece and in her MA thesis she explored word order in GSL in simple declarative sentences and in wh-questions. In her PhD project, she investigates imperative speech acts in Greek Sign Language (GSL) and German Sign Language (DGS) by using elicitation and judgement tasks. Through her project she aims to discover which strategies -morphosyntactic and prosodic- are employed in sign languages for the articulation of imperative speech acts and define the contribution of particular manual and non-manual elements. One of her research goals is to find whether sign languages possess a particular sentence type for the expression of directive constructions, similar to the one we call “Imperative” in spoken languages, and how it can be defined by applying specific diagnostics.
Rui Liu
(Associate Member)
Ghent University
rui.liu@ugent.be
Rui Lui works as a postdoc researcher at Ghent University (PI: Lara Bardi). Rui studies human communication with eye/motion-tracking, M/EEG, fMRI, psychopharmacology, and social network analysis. She is now combing experimental semiotic paradigms and EEG hyperscanning to examine behavioral and neural mechanisms underlying real-time interpersonal adjustment during conventionalization of partner-specific communication system. She obtained her PhD at Donders Center for Cognitive Neuroimaging under the supervision of Ivan Toni. During her PhD study, she was investigating perception and production of communicative action. She is also working with Wim Pouw at Dynamic Signs and Signals group (DYNAMOS) at Donders, where they are investigating distinct manual kinematic signatures that speakers and signers used for communication.
Selected publications
- Liu, R., Bardi, L., (2024) Interpersonal synergy in co-constructing shared conceptual space. 7th European Society for Cognitive and Affective Neuroscience (ESCAN): Ghent, Belgium.
- Liu, R,. Pouw, W., Goldin-Meadow, S., Brentari, D. (preprint) Signers and speakers show distinct temporal kinematic signatures in their manual communicative movements. PsyArXiv.
- Liu, R., Stolk, A., de Boer, M., Oostenveld, R. & Toni, I. (preprint) Oxytocin facilitates communicative adjustment by upregulating broadband aperiodic neural activity. PsyArXiv. doi: 10.31234/osf.io/kq9g6
- Liu, R., Bögels, S., Bird, G., Medendorp, W.P. & Toni, I. (2021) Hierarchical Integration of Communicative and Spatial Perspective-Taking Demands in Sensorimotor Control of Referential Pointing. Cognitive Science. 46(1). doi: 10.1111/cogs.13084
- Liu, R., Yuan, X., Chen, K., Jiang, Y. & Zhou. W. (2018) Perception of social interaction compresses subjective duration in an oxytocin-dependent manner. eLife 7:e32100.
Matteo Maran
(Associate Member)
Donders Center for Cognition, Radboud University Nijmegen
matteo.maran@donders.ru.nl
Matteo Maran is a Postdoctoral Researcher at the Donders Center for Cognition (DCC) at Radboud University (Nijmegen, The Netherlands). He is a member of the Speech Perception in Audiovisual Communication (SPEAC) research group, funded by the ERC Starting Grant ‘HearingHands’ (101040276; PI: Hans Rutger Bosker) that started in September 2022. He works on the audiovisual integration of gestural timing with spoken prosody in neurotypical and autistic individuals. Matteo Maran has a background in cognitive psychology, neuroscience and psycholinguistics, obtained during his Bachelor’s and Master’s degrees at the University of Padova (Padova, Italy) and his PhD project at the Max Planck Institute for Human Cognitive and Brain Sciences (Leipzig, Germany).
Selected publications
- Maran, M., Friederici, A. D., & Zaccarella, E. (2022). Syntax through the looking glass: A review on two-word linguistic processing across behavioral, neuroimaging and neurostimulation studies. Neuroscience & Biobehavioral Reviews, 142, 104881. https://doi.org/10.1016/j.neubiorev.2022.104881
- Maran, M., Numssen, O., Hartwigsen, G., & Zaccarella, E. (2022). Online neurostimulation of Broca’s area does not interfere with syntactic predictions: A combined TMS-EEG approach to basic linguistic combination. Frontiers in Psychology, 13, 968836. https://doi.org/10.3389/fpsyg.2022.968836
- Pyatigorskaya*, E., Maran*, M., & Zaccarella, E. (2023). Testing the automaticity of syntax using masked visual priming. Language, Cognition and Neuroscience, 1–25. https://doi.org/10.1080/23273798.2023.2173790
- Van Der Burght*, C. L., Friederici*, A. D., Maran*, M., Papitto*, G., Pyatigorskaya*, E., Schroen*, J., Trettenbrein*, P. C., & Zaccarella*, E. (2022). Cleaning up the Brickyard: How Theory and Methodology Shape Experiments in Cognitive Neuroscience of Language [Preprint]. PsyArXiv. https://doi.org/10.31234/osf.io/6zpjq
Chiara Marchetiello
(Associate Member)
The GestuGram Lab, Trinity College Dublin
marchtetc@tcd.ie
Chiara Marchetiello is a PhD student in Linguistics at Trinity College Dublin. Her doctoral project The Syntax of Gestures in the Local Languages of Campania is funded by Trinity’s Provost’s PhD Project Award awarded to the research project Gestural Grammar: Investigating Gestures in Southern Italy (GestuGram) led by Prof Valentina Colasanti. Chiara’s dissertation aims to provide new insights on the grammatical integration of gestures using as empirical base the gestural inventories of the languages spoken in Campania region (e.g., Neapolitan, Torrese, Acerrano, etc.).
Chiara is primarily interested in theoretical syntax and its interfaces with semantics and phonology, with particular focus on Romance languages (especially the Italo-Romance subfamily of languages). Her research interests include negation and negative concord, the structure of nominals and vocatives, epistemic modality, and discourse markers.
Selected publications (seminar and conference talks, peer-reviewed abstracts)
- In press. Colasanti, Valentina and Chiara Marchetiello. Gesture and information structure: a case study on topic markers in Southern Italo-Romance. Italian Journal of Linguistics.
- 2025. Gestures of negation: True or false? Talk to be given at the Formal Linguistics Approaches to Multimodality (FLAMM), Trinity College Dublin, 4th December.
- 2025. Co-speech [PDO] is an epistemic gestural marker in Campania: end of discussion! Talk given at the Linguistics Research Seminars, Trinity College Dublin, 11 November 2025.
- 2025. Parasitic licensing in the visual-gestural modality, Annual Meeting of the Linguistic Association of Great Britain, 4 September 2025, University of Suffolk, UK.
- 2025. Some initial thoughts on the syntax of gestures in the languages of Campania. Talk given at the Romance Linguistics Circle (RoLinC), University of Cambridge, 11 March 2025.
Josiah Nii Ashie Neequaye
(Associate Member)
University of Göttingen
josiah.neequaye@uni-goettingen.de
Josiah Nii Ashie Neequaye works on iconicity, with special focus on ideophones and their interaction with other iconic enrichments such as gestures. He completed his Bachelor’s degree in Economics and Linguistics, and his Master’s degree in Linguistics, both at the University of Ghana. He is a doctoral researcher in the RTG 2636: “Form-Meaning Mismatches” at the University of Göttingen. His PhD project generally explores the interaction between ideophones and co-speech gestures in Ga (Kwa, Niger-Congo), and how these (conventionalized) ideophone-accompanying gestures compare with ideophone-like expressions in sign language.
Selected publications
- Asiedu, Prince, Mavis Boateng Asamoah, Kathryn Barnes, Reginald Duah, Cornelia Ebert, Josiah Nii Ashie Neequaye, Yvonne Portele, and Theresa Stender. “On the information status of ideophones in Akan.” In Annual Conference of African Languages, vol. 45. 2023.
Babajide Owoyele
(Associate Member)
Erasmus School of Social and Behavioural Sciences, Rotterdam, The Netherlands & Hasso Plattner Institute, Potsdam
owoyele@drift.eur.nl, babajide.owoyele@hpi.de
Babajide Owoyele is a design scientist who uses multimodal analysis to explore challenges at the intersection of computer science and innovation research. He is a PhD candidate at the Erasmus School of Social and Behavioural Sciences (Prof Derk Loorbach) and a research associate at the Hasso Plattner Institute, supervised by Prof Gerard de Melo at the Artificial Intelligence and Intelligence Systems Group. His thesis at DRIFT explores leveraging design science and multimodal data to map triadic collaboration dynamics in multi-actor constellations at the (1) macro-level (inter-organizational networks) and at the micro level (teams engage in ideation). As part of the Envisionbox.org team, he is exploring how to make audio-visual data archiving and analytics more accessible to early career researchers and data stewards.
Babajide earned his Master’s degree in Global Production Engineering (Solar Technology) from the Technical University of Berlin in 2017, alongside a degree in Climate Knowledge and Systemic Innovation from the European Institute of Innovation and Technology (EIT). His research and professional experience include positions at the Wuppertal Institute, ESA BIC, and several startups. Babajide holds a Bachelor’s degree in Electrical Engineering from Covenant University, Ota, Nigeria.
Selected publications
- Babajide Alamu Owoyele, Martin Schilling, Rohan Sawahn, Niklas Kaemer, Pavel Zherebenkov, Bhuvanesh Verma, Wim Pouw, Gerard de Melo (preprint (2024)). “MaskAnyone Toolkit: Offering Strategies for Minimizing Privacy Risks and Maximizing Utility in Audio-Visual Data Archiving.” arXiv. https://doi.org/10.48550/arXiv.2408.03185
- Babajide Owoyele, Bhuvanesh Verma, Victor Omolaoye, Jonathan Antonio Edelman, Derk Loorbach, Gerard de Melo (preprint (2024)). “Socio-Semantic X-Ray of Multi-Actor Constellations using Topics and Interstitial Authors: A Toolkit for Augmenting Computational Literature Reviews.” SSRN. https://dx.doi.org/10.2139/ssrn.4713155
- Babajide Owoyele, James Trujillo, Gerard de Melo, Wim Pouw. “Masked-Piper (2022). Masking personal identities in visual recordings while preserving multimodal information.” SoftwareX, Volume 20, 101236. https://doi.org/10.1016/j.softx.2022.101236
Lara Pearson
(Associate Member)
University of Cologne
lpearson@uni-koeln.de
Lara Pearson is Professor of Ethnomusicology at the Institute of Musicology, University of Cologne. Her work explores gesture, vocalization and interaction in music-related contexts, with a focus on the performance and pedagogical practices of Karnatak vocalists in South India. Her research is highly interdisciplinary, combining methods from gesture studies, ethnomusicology, computational musicology and human movement science. In addition to her work on music and gesture, she has published on coarticulation in music, cross-cultural aesthetics, music notation and concepts of improvisation. She is currently Vice-Chair of the International Council for Traditions of Music and Dance Study Group on Sound, Movement and the Sciences (ICTMD SoMoS).
- Pearson, L. (in press). Adjusting to the Other: Audiencing Through Gesture in Karnatak Vocal Lessons. Journal of the Royal Musical Association.
- Pearson, L., & Pouw, W. (2022). Gesture–Vocal Coupling in Karnatak Music Performance: A Neuro–bodily Distributed Aesthetic Entanglement. Annals of the New York Academy of Sciences, 1515(1), 219–236. https://doi.org/10.1111/nyas.14806
- Pearson, L. (2022). Inscription, Gesture, and Social Relations: Notation in Karnatak Music. In E. Payne & F. Schuiling (Eds.), Material Cultures of Music Notation: New Perspectives on Musical Inscription (pp. 139–151). Abingdon: Routledge. https://doi.org/10.4324/9780429342837-13
- Pearson, L. (2016). Coarticulation and Gesture: An Analysis of Melodic Movement in South Indian Raga Performance. Music Analysis, 35 (3), 280-313. https://doi.org/10.1111/musa.12071
- Pearson, L. (2013). Gesture and the Sonic Event in Karnatak Music. Empirical Musicology Review, 8 (1), 2-14. https://doi.org/10.18061/emr.v8i1.3918
Vanessa Wing Yan Tsang
(Associate Member)
University of Göttingen
wingyan.tsang@uni-goettingen.de
Vanessa Tsang works on spatial constructions and iconicity in the visual modality, with a major focus on sign languages. She completed her BA studies in Linguistics at the Chinese University of Hong Kong and then her MA in Linguistics at the University of Cologne. Currently, she is a doctoral student of the RTG 2636 “Form-meaning mismatches” at the University of Göttingen. Her project compares event description of motion and speed in Hong Kong Sign Language (HKSL) and German Sign Language (DGS), with the aim of investigating the language-specific properties and sign-gesture interface of classifier constructions. At a later stage, gestural productions are included for a more holistic examination of the modality-specific features of motion representations in the visual modality.
ViCom Junior Fellow
Dr. Vinicius Macuch Silva
(ViCom Junior Fellow)
Goethe University Frankfurt
vini.macuch@gmail.com
Vinicius Macuch Silva is interested in how people use language to create meaning in communication, both when interacting with one another and when producing and interpreting language in various other settings. In his research he primarily uses quantitative empirical methods, including controlled experimentation as well as computational and corpus analysis, to investigate issues related to pragmatic inferencing, strategic communication, argumentation and stance-taking, as well as expressive and affective meaning.
Selected publications
- Macuch-Silva, V., Lorson, A., Franke, M., Cummins, C., & Winter, B. (2024). Strategic use of English quantifiers in the reporting of quantitative information. Discourse Processes. https://doi.org/10.1080/0163853X.2024.2413311
- Lorson, A., Macuch-Silva, V., Hart, C., & Winter, B. (2024). Gesture size affects numerical estimates in quantifier comprehension. Journal of Experimental Psychology: Learning, Memory, and Cognition. https://dx.doi.org/10.1037/xlm0001372
- Macuch-Silva, V., & Franke, M. (2021). Pragmatic Prediction in the Processing of Referring Expressions Containing Scalar Quantifiers. Frontiers in Psychology. https://doi.org/10.3389/fpsyg.2021.662050
- Macuch-Silva, V., Holler, J., Özyürek, A., & Roberts, S. (2020). Multimodality and the origin of a novel communication system in face-to-face interaction. Royal Society Open Science. https://doi.org/10.1098/rsos.182056
- Jamieson, E, & Macuch-Silva, V. (joint first authors, to appear). Psycholinguistic processing tasks and the study of question bias. In Benz, A., Krifka, M., Trinh, T. and Yatushiro, K. (eds.) Biased Questions: Experimental Results and Theoretical Modelling.