Project Participants
PD. Dr. Oliver Herbort
(Principal Investigator)
University of Würzburg
oliver.herbort@psychologie.uni-wuerzburg.de
Oliver Herbort is interested in human motor control, embodied choices, and the production and perception of pointing gestures. Specifically, he tries to understand the relationship between the body postures assumed during pointing, the objects or locations that are perceived to be indicated, and various situational factors. His methods include motion capture, virtual reality setups, and formal models. Oliver Herbort received his PhD from the University of Würzburg, was a post-doc in Würzburg and Tübingen, and is now an academic counsel at the University of Würzburg and PI of the VICOM project “representation of pointing uncertainty for the integration of pointing gestures and speech”.
Selected publications
- Herbort, O., Krause, L.-M., & Kunde, W. (2021). Perspective determines the production and interpretation of pointing gestures. Psychonomic Bulletin & Review, 28(2), 641-648. https://doi.org/10.3758/s13423-020-01823-7
- Herbort, O. & Kunde, W. (2016). Spatial (mis-)interpretation of pointing gestures to distal referents. Journal of Experimental Psychology: Human Perception and Performance, 42(1), 78-89. https://doi.org/10.1037/xhp0000126
- Herbort, O., & Kunde, W. (2018). How to point and to interpret pointing gestures? Instructions can reduce pointer-observer misunderstandings. Psychological Research, 82(2), 395-406. https://doi.org/10.1007/s00426-016-0824-8
- Krause, L.-M., & Herbort, O. (2021). The observer’s perspective determines which cues are used when interpreting pointing gestures. Journal of Experimental Psychology: Human Perception and Performance, 47(9), 1209-1225. https://doi.org/10.1037/xhp0000937
Lisa-Marie Krause
(PhD Candidate)
University of Würzburg
lisa-marie.krause@uni-wuerzburg.de
Lisa-Marie started her PhD in 2019 at the University of Würzburg. In her research, she is mainly interested in the spatial interpretation of pointing gestures, especially how different factors, e.g., perspective or arm posture, are affecting this interpretation. Within ViCom, she will extend this focus by gesture-speech integration and pointing production. She aims to examine how spatial information that are expected to be communicated by pointers and the spatial information observers expected to be implied by pointing gestures match. Methodically, Lisa-Marie conducts her experiments using eye tracking in virtual reality as well as in real-life dyadic interactions.
Selected Publications
- Krause, L.-M., & Herbort, O. (2021). The observer’s perspective determines which cues are used when interpreting pointing gestures. Journal of Experimental Psychology: Human Perception and Performance, 47(9), 1209-1225. http://dx.doi.org/10.1037/xhp0000937
- Herbort, O., Krause, L.-M., & Kunde, W. (2021). Perspective determines the production and interpretation of pointing gestures. Psychonomic Bulletin & Review, 28(2), 641-648. https://doi.org/10.3758/s13423-020-01823-7
- Herbort, O. & Krause, L.-M. (2023). The efficiency of augmented pointing with and without speech in a collaborative virtual environment. In: V. G. Duffy (Ed.) Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management. HCII 2023. Lecture Notes in Computer Science, vol 14028. (pp. 510-524). Springer. https://doi.org/10.1007/978-3-031-35741-1_37
- Krause, L.-M. & Herbort, O. (2023). Just visual context or part of the gesture? The role of arm orientation in bent pointing interpretation. Acta Psychologica, 241, 104062. https://doi.org/10.1016/j.actpsy.2023.104062
Project Description
Pointing gestures are ubiquitous in every-day life. Typically, pointing is used to direct the attention of others to distant objects in the environment, especially when the objects are difficult to describe verbally (e.g., a star in the night sky or a specific chocolate at a pastry shop). Helpful as pointing may be, pointers do not expect that the pointing gesture alone establishes a joint focus of attention. Hence, pointing is typically complemented by speech. To choose a meaningful verbal description, a pointer needs some notion about the uncertainty associated with his gesture. That is, if a pointer assumes that his gesture may direct another person’s attention exactly to the referent, fewer verbal descriptions (if any) are given than when he expects the point to only draw attention to the rough vicinity of the referent. Likewise, observers of pointing gestures represent their own perceptual uncertainty to know in which area to look for the referent or to judge the confidence of their guess. In summary, effective communication with pointing gestures and speech requires representations of perceptual uncertainty on the sides of the pointers and observers. However, little is known about the specific information pointers assume to convey with pointing, observer assume can be inferred from a point, and how they relate to the information that observers factually infer from pointing gestures. The objective of the present proposal is to examine such representations of uncertainty for the typical case of pointing to small and distant objects. To examine representations of uncertainty, we plan to use visual search tasks, in which the identity of a target is only known to a pointer, who communicates it to an observer with pointing and speech. Eye-tracking data is used to derive representations of uncertainty as pointers and observers need to visually process an area of uncertainty around the (assumed) referent to provide adequate verbal descriptions or when searching for the referent. Additionally, representations of uncertainty are derived from the specificity of the pointer’s verbal descriptions and the level of specificity needed by the observer to be confidence about their guess. The work programme includes experiments in real and virtual settings. WP 1 aims at establishing the basic paradigm. WP 2 addresses whether representations of uncertainty reflect factors that are known to affect pointing perception. WP 3 addresses how representations of uncertainty may be adapted to perceptual performance. The results of WP 1 – 3 are used to extent our formal model of pointing perception to include representations of uncertainty. WP 4 evaluates methods to improve pointing-based communication by making misconceptions about uncertainty explicit in real and virtual settings. In summary, we expect that the project provides new insights into representations of the uncertainty of pointing perception and their effect on the integration of gestures and speech.