Researchers involved: Oliver Herbort, Lisa-Marie Krause (Representation of Pointing Uncertainty for the Integration of Pointing Gestures and Speech), Cornelia Ebert, Stefan Hinterwimmer, Sebastian Walter (Visual and Non-visual Means of Perspective Taking in Language)
In the proposed short-term collaboration, we address whether reported vs. direct speech affects how concurrent pointing gestures are interpreted, thus combining topics of the involved ViCom projects. To illustrate the research question, consider the scene below: Person A (depicted in the scene) reports some previous actions of another person B (not present anymore). In the scene, A’s pointing gesture is ambiguous. One interpretation of A’s pointing gesture is that B meant the safe on the right. However, it is also possible to infer that A “quotes” B’s pointing gesture and that B, who stood in the door on the right and looked to the left, pointed to the safe on the left.
We hypothesized that the type of speech accompanying the pointing gesture is used to disambiguate the situation. If A uses direct speech, as in the example, it is more likely that A’s pointing gesture is considered a gestural quotation, which is not directly interpreted but transformed to the viewpoint of the quoted person B. By contrast, if A uses reported speech, a direct interpretation of the gesture is more likely.
We already conducted an online experiment, in which a person reported the action of another unseen person to the participants. The report was presented in comic-like scenes (the image above is adapted from the experiment). After the report, participants were asked to identify which safe was meant and judge the certainty of their response.
The experiment had three main results. First, participants can transform a pointing gesture to the viewpoint of an unseen person (if a verbal report of the pointing gesture is given). Second, there is a strong tendency to directly interpret pointing gestures, irrespective of whether the report is given in direct or reported speech. Third, we found a small to medium effect (d = .32) of speech report type, which approached significance (p = .070).
To summarize, we have an experimental paradigm that is in principle suitable to elicit the hypothesized effect, but the experimental power of the experiment did not suffice to reach a decision about our hypothesis. The aim of the proposal is to fund two follow-up experiments to complete this yet informal collaboration.