Visual and Non-visual Means of Perspective Taking in Language

Project Participants

Project Description

Perspective is a crucial aspect of the information conveyed by linguistic utterances, and languages provide various means to signal whether an utterance reflects the perspective of the speaker or narrator, or that of some discourse referent. Common methods of conveying a discourse referent’s perspective include Direct Discourse, Indirect Discourse, Free Indirect Discourse, Protagonist Projection, and Viewpoint Shifting. The syntactic, semantic, and pragmatic properties of these methods have been extensively studied (see, e.g., Schlenker 2004; Sharvit 2008; Stokke 2013; Eckardt 2014; Maier 2015; Hinterwimmer 2017, 2019; Abrusán 2020). Additionally, it is well known that certain linguistic markers can signal the prominence of the speaker’s or narrator’s perspective, such as the use of deictic expressions, speech act particles, and modal verbs (see Zeman 2019 for a recent overview), as well as the use of demonstrative pronouns to refer to topical protagonists (Hinterwimmer, Brocher & Patil 2020; Hinterwimmer 2020).

Furthermore, perspective taking can also be expressed through co-speech gestures, i.e., gestures that speakers produce while uttering sentences (see Ebert & Ebert 2014; Schlenker 2018; Ebert, Ebert & Hörnig 2020 for recent analyses of co-speech gestures in the formal semantic realm). Specifically, two types of iconic gestures often reveal perspective: character viewpoint gestures (CVGs) and observer viewpoint gestures (OVGs) (McNeill 1992; Parrill 2010, 2012; Stec 2012, 2016). When performing CVGs, the speaker impersonates an individual participating in the event described by the sentence, enacting the event from that individual’s point of view using their entire body and facial expressions. In contrast, when performing OVGs, the speaker depicts the event as if viewed from a distance, typically using only their hands to represent a participant, with the hand’s trajectory illustrating that participant’s path.

However, the interaction between linguistic and gestural perspective taking has not been systematically investigated (though see Hinterwimmer, Patil & Ebert 2021; Ebert & Hinterwimmer 2022 for initial explorations). This project aims to deepen our understanding of how these two types of perspective taking interact. Specifically, we intend to address the following questions through experimental methods such as acceptability rating and forced-choice studies: Is there a preference for alignment between linguistic and gestural perspectives? If so, how strong is this preference, and under what conditions can it be overridden? Does this preference differ across various types of perspective taking? Based on the experimental findings, we will develop a theoretical model to explain how perspectival information on both levels integrates to convey a single, coherent message. This research will not only advance our understanding of perspective taking but also shed light on the interaction between speech and co-speech gestures more broadly.

Project Activities

Publications

Ebert, C. & S. Walter. To appear. Expressivity and gestures. To appear in Gutzmann, Daniel & Katharina Turgay (eds.), The Oxford Handbook of Expressivity in Language. Oxford, UK & New York, NY: Oxford University Press.

Ebert, Cornelia, Stefan Hinterwimmer, Markus Steinbach & Sebastian Walter. Forthcoming. Formal semantics on visual communication. In John A. Bateman & Chiao-I Tseng (eds.), Handbook of Linguistics and Multimodality. Chichester, UK: Wiley.

Ebert, C., E. Maier & M. Steinbach. Under review. Visual components to interpretation – formal approaches to visible meaning. Submitted to Linguistics Compass.

Walter, Sebastian. 2025. Indirect discourse as mixed quotation? An experimental investigation. In Tyler Knowlton, Florian Schwarz & Anna Papafragou (eds.), Proceedings of Experiments in Linguistic Meaning (ELM) 3, 423–434. Philadelphia, PA: UPenn. doi:10.3765/elm.3.5845.

Walter, Sebastian & Stefan Hinterwimmer. 2025. An experimental investigation of perspective alignment in gesture and speech. In Tyler Knowlton, Florian Schwarz & Anna Papafragou (eds.), Proceedings of Experiments in Linguistic Meaning (ELM) 3, 411–423. Philadelphia, PA: UPenn. doi:10.3765/elm.3.5757.

Walter, Sebastian. 2025. A mixed-quotational account of indirect discourse: Evidence from self-pointing gestures. In Yao Zhang, Fengyue (Lisa) Zhao, Youngdong Cho & Yifan Wu (eds.), Proceedings of Semantics and Linguistic Theory (SALT) 34, 88–109. Rochester, NY: University of Rochester. doi:10.3765/xxnepg45.

Walter, Sebastian, Cornelia Ebert & Stefan Hinterwimmer. 2025. Viewpoint matters: Prototypical vs. non-prototypical gestures in the VP domain. In Federica Longo & Daniela Panizza (eds.), Proceedings of Sinn und Bedeutung (SuB) 29, 1646–1661. Noto, Italy: University of Messina. doi:10.18148/sub/2024.v29.1300.

Walter, Sebastian. 2025. Investigations on the at-issue status of viewpoint gestures. Linguistische Berichte 283, 319–352. doi:10.46771/9783967699494_3.

Walter, Sebastian & Yvonne Portele. 2025. Pronoun interpretation in German speech reports. In Antonis Botinis (ed.), Proceedings of International Conference of Experimental Linguistics (ExLing) 15, 137–140. Paris, France: Université Paris Cité.

Ecsedi, Noémi, Cornelia Ebert, Kurt Erbach & Sebastian Walter. 2025. Can character viewpoint gestures guide pronoun resolution in German? In Antonis Botinis (ed.), Proceedings of International Conference of Experimental Linguistics (ExLing) 15, 33–36. Paris, France: Université Paris Cité.

Walter, Sebastian & Stefan Hinterwimmer. 2025. Shifted face emoji in indirect discourse: A mixed-quotational analysis. In Federica Longo & Daniela Panizza (eds.), Proceedings of Sinn und Bedeutung (SuB) 29, 1662–1680. Noto, Italy: University of Messina. doi:10.18148/sub/2024.v29.1301.

Walter, Sebastian. 2024. The at-issue status of viewpoint gestures: Evidence for gradient at-issueness. In Geraldine Baumann, Daniel Gutzmann, Jonas Koopman, Kristina Liefke, Agata Renans & Tatjana Scheffler (eds.), Proceedings of Sinn und Bedeutung (SuB) 28, 943–960. Bochum, Germany: Ruhr University Bochum.

Ebert, C., G. Pirillo & S. Walter. 2022. The role of gesture-speech alignment for gesture interpretation. In Sam Featherston, Robin Hörnig, Andreas Konietzko & Sophie von Wietersheim (eds.), Proceedings of Linguistic Evidence 2020: Linguistic theory enriched by experimental data, 65–77. Tübingen, Germany: University of Tübingen.

Ebert, C. & S. Hinterwimmer. 2022. Free indirect discourse meets character viewpoint gestures. In S. Featherston, R. Hörnig, A. Konietzko & S. von Wietersheim (eds.), Proceedings of Linguistic Evidence 2020: Linguistic theory enriched by experimental data, 333–349. Tübingen, Germany: University of Tübingen.

Conference contributions

Walter, Sebastian & Lennart Fritzsche. Head nods don’t always mean yes: Ambiguity in gestural responses to negative questions. Poster at Architectures and Mechanisms for Language Processing (AMLaP) 31, Prague, Czech Republic. (September)

Portele, Yvonne & Sebastian Walter. Pronoun interpretation in German speech reports. Poster at Architectures and Mechanisms for Language Processing (AMLaP) 31, Prague, Czech Republic. (September)

Fritzsche, Lennart & Sebastian Walter. Nodding yes or doch? On the interpretation of gestural response elements. Poster at LingCologne 2025, Cologne, Germany. (May)

Krause, Lisa-Marie, Oliver Herbort, Cornelia Ebert, Stefan Hinterwimmer & Sebastian Walter. Interpreting pointing gestures: The role of direct and indirect speech. Poster at Tagung experimentell arbeitender Psychologen (TeaP) 2025, Frankfurt, Germany. (March)

Walter, Sebastian & Stefan Hinterwimmer. Shifted face emoji in indirect discourse: A mixed-quotational approach. Talk at the workshop Quotation unbound – Bridging the divide across quotation types at the Annual Conference of the German Linguistics Society 47, Mainz, Germany. (March)

Walter, S. & Y. Portele. 2024. Pronoun interpretation in German speech reports. Talk at International Conference of Experimental Linguistics (ExLing) 15, Paris, France. (October)

Ecsedi, N., C. Ebert, K. Erbach & S. Walter. 2024. Can character viewpoint gestures guide pronoun resolution in German?. Poster at International Conference of Experimental Linguistics (ExLing) 15, Paris, France. (October)

Walter, S. 2024. Differences in the at-issue status of viewpoint gestures: Inherent or QUD-dependent?. Talk at International Congress of Linguists (ICL) 21, Poznan, Poland. (September)

Walter, S., C. Ebert & S. Hinterwimmer. 2024. An experimental investigation of perspective alignment in gesture and speech. Poster at Architectures and Mechanisms for Language Processing (AMLaP) 30, Edinburgh, UK. (September)

Walter, S. 2024. Can face emoji receive a shifted interpretation in indirect discourse?. Poster at Architectures and Mechanisms for Language Processing (AMLaP) 30, Edinburgh, UK. (September)

Walter, S., C. Ebert & S. Hinterwimmer. 2024. Viewpoint matters: Prototypical vs. non-prototypical co-speech gestures in the VP domain. Poster at Sinn und Bedeutung (SuB) 29, Noto, Italy. (September)

Walter, S. & S. Hinterwimmer. 2024. Shifted face emoji in indirect discourse: A mixed-quotational approach. Talk at Sinn und Bedeutung (SuB) 29, Noto, Italy. (September)

Barnes, K. & S. Walter. 2024. Ideophones and reported speech: Depicting actions, events and speech. Talk at Sinn und Bedeutung (SuB) 29, Noto, Italy. (September)

Walter, S. 2024. Indirect discourse as mixed quotation: Evidence from self pointing gestures. Short talk at Experiments in Linguistic Meaning (ELM) 3, Philadelphia, PA. (June)

Walter, S. & S. Hinterwimmer. 2024. An experimental investigation of perspective alignment in gesture and speech. Short talk at Experiments in Linguistic Meaning (ELM) 3, Philadelphia, PA. (June)

Walter, S. 2024. Indirect discourse as mixed quotation: Evidence from self pointing gestures. Poster at Semantics and Linguistic Theory (SALT) 34, Rochester, NY. (May)

Walter, S. 2023. The at-issue status of viewpoint gestures: Evidence for gradient atissueness. Talk at Sinn und Bedeutung 28, Bochum, Germany. (September)

Walter, S., C. Ebert & S. Hinterwimmer. 2023. Are there salience differences between viewpoint gestures? Poster at XPrag X, Paris, France. (September)

Walter, S. 2022. The (not-)at-issue status of character viewpoint gestures. Short talk at Linguistic Evidence 2022, Paris, France. (October)

Workshop presentations

Walter, Sebastian & Cornelia Ebert. 2025. Adding to the expressive typology: A formal analysis of emblematic gestures as visual expressives. Talk at EPITHETS & STAL-2025 Workshop, Genoa, Italy. (May)

Walter, Sebastian & Lennart Fritzsche. 2025. A demonstrational analysis of modified pro-speech gestures. Talk at Dimensions of Iconicity in the Visual Modality, Göttingen, Germany. (February)

Herbort, O., L.-M. Krause, C. Ebert, S. Hinterwimmer & S. Walter. 2024. Pointing in direct and indirect speech. Talk at the ViCom Workshop The Linguistic Status of Demonstrations, Königstein im Taunus, Germany. (April)

Walter, S. 2024. Indirect discourse as mixed quotation: Evidence from self pointing gestures. Talk at the ViCom Workshop The Linguistic Status of Demonstrations, Königstein im Taunus, Germany. (April)

Ebert, Cornelia, L. Fritzsche, W. Pouw, M. Steinbach & S. Walter. 2024. Multimodal response strategies to polar questions in German. Talk at the ViCom Workshop The Linguistic Status of Demonstrations, Königstein im Taunus, Germany. (April)

Walter, S., C. Ebert & S. Hinterwimmer. 2023. Is there perspective alignment in gesture and speech? Talk at ViCom Annual Meeting 2023, Bad Homburg, Germany. (November)

Walter, S., C. Ebert & S. Hinterwimmer. 2023. Pointing gestures in reported speech. Talk at ViCom Theory Workshop 2023, Königstein im Taunus, Germany. (April)

Walter, S., C. Ebert & S. Hinterwimmer. 2022. Visual and non-visual means of perspective taking in language. Talk at ViCom Annual Meeting 2022, Bad Homburg, Germany. (November)

Invited talks and guest lectures

Ebert, C. 2023. A cross-modal account of dynamic binding in gesture-speech interaction. Invited talk at the special session The semantics and pragmatics of co-speech/co-sign communication at Sinn und Bedeutung (SuB) 28, Bochum, Germany. (September)

Walter, S. 2023. The at-issue status of viewpoint gestures: Evidence for gradient atissueness. Invited talk at Bochum Language Colloquium, Bochum, Germany. (October)

Walter, S. 2024. Indirect discourse as mixed quotation: Evidence from self pointing gestures. Invited talk at MorphoSynSem Colloquium, Potsdam, Germany. (May)

Walter, S. 2024. Perspektive in Sprache und Geste (Perspective in gesture and speech). Guest lecture in the seminar Perspektive in der Sprache (instructor: Stefan Hinterwimmer). Hamburg, Germany: University of Hamburg. (June)

Summer school courses

Ebert, C. & M. Steinbach. 2023. The semantics of visual communication. Theoretical approaches to visual meaning aspects of co-speech gestures and sign language. Course taught at European Summer School in Logic, Language and Information (ESSLLI) 34. (August)

Events
DateEventOrganization
June 14, 2023Closing Workshop of the seminar Visual communicationCornelia Ebert, Stefan Hinterwimmer & Markus Steinbach
September 23, 2025Special session Beyond descriptive meaning: The semantics and pragmatics of demonstrations of Sinn und Bedeutung (SuB) 30Kathryn Barnes, Cornelia Ebert, Lennart Fritzsche, Markus Steinbach &Sebastian Walter
September 23-September 27, 2025Sinn und Bedeutung (SuB) 30Kathryn Barnes, Cornelia Ebert, Kurt Erbach, Lennart Fritzsche, Janek Guerrini, Cécile Meier, Carolin Reinert, Frank Richter, Manfred Sailer, Theresa Stender, Sebastian Walter, Eleonora Zani & Alexandra Zinke
Short-term Collaborations

A crossmodal investigation of negation and rejection II
Sebastian Walter & Lennart Fritzsche (external collaborator) (2025)

The effect of reported and direct speech on perspective taking when interpreting concurrent pointing gestures
Oliver Herbort, Cornelia Ebert, Stefan Hinterwimmer, Lisa-Marie Krause & Sebastian Walter (2025)

A crossmodal investigation of negation and rejection. Sebastian Walter & Lennart Fritzsche (external collaborator) (2024)

Interactions of at-issueness and commitment in lying: Can iconic gestures be used to lie?. Jonas Hartke (external collaborator) & Sebastian Walter (2023)

Outreach

Ebert, C. & M. Steinbach. 2024. Die sprechenden Hände. Über den Unterschied zwischen Wörtern, Gesten und Gebärden (Speaking hands. On the difference between words, gestures, and signs). Talk at 15. WissensWerkstatt für Kinder. (November)

Hinterwimmer, S. 2024. Alltagskraft der Gesten (Everyday Power of Gestures). TV interview in the WDR format Hier und Heute, Link. (April)

Hinterwimmer, S. 2024. Mit Gesten besser kommunizieren und überzeugen (Communicating and Persuading Better with Gestures). TV Interview in the WDR format Lokalzeit Bergisches Land, Link. (March)

Ebert, C. & M. Steinbach. 2023. Die sprechenden Hände. Über den Unterschied zwischen Wörtern, Gesten und Gebärden (Speaking hands. On the difference between words, gestures, and signs). Talk at 20th Children’s University at Goethe University Frankfurt. (October)