How do language, gestures, and images express perspective together—or differently? This project investigates how meaning is shaped across spoken language, gesture, emojis, memes, and other visual demonstrations in multimodal communication. Combining formal semantics with experimental methods, the research examines which visual elements carry meaning and how they express stance, emotion, or viewpoint. The findings will advance our understanding of perspective taking and contribute to the study of visual communication across modalities.
Project Members
Cornelia Ebert
Principal Investigator
Goethe-University, Germany
Stefan Hinterwimmer
Principal Investigator
University of Hamburg, Germany
Sebastian Walter
Postdoc
Goethe-University, Germany
Daniel Teich
Phd Candidate
University of Hamburg, Germany
Project Description
The project “Visual and non-visual means of perspective taking in language II” explores how perspective can be expressed simultaneously and divergently across language, gesture, and image in multimodal communication. Building on findings from the first project phase—which showed that gesture and speech are not necessarily aligned in terms of perspective—the second phase aims to develop a formal theory of so-called demonstrations (e.g., gestures, emojis, memes). The core assumption is that demonstrations are inherently perspective-dependent.
The central focus lies on how gestures and images contribute iconic information, which aspects of these demonstrations are actually meaning-bearing (depictive), and how they interact with spoken or written language. The project further develops theoretical distinctions, such as those between prototypical and non-prototypical gestures or between object-related and concept-related interpretations. In parallel, it investigates how visual content in combination with language—for example in political memes—is interpreted, which visual features are taken to be relevant, and how they contribute to conveying stance or emotion.
The project consists of four work packages: (1) investigates perspective taking in gestures, (2) in text-image combinations like memes, (3) develops a formal semantics of demonstrative acts, and (4) addresses experimentally when visual elements are interpreted as central, i.e., at-issue. A special focus is placed on improving diagnostic tools for identifying meaning-contributing content in the visual modality—for example, through reaction time experiments comparing linguistic and gestural material.
The aim is to develop an integrated, theoretically grounded and empirically supported model of multimodal perspective taking. In doing so, the project makes an innovative contribution to formal semantics, experimental pragmatics, and the interdisciplinary study of visual communication.