Integration of visual and spoken cues in a virtual reality navigation task

When integrating information in real time from multiple modalities or sources, such as when navigating with the help of GPS voice instructions along with a visual map, a decision-maker is faced with a difficult cue integration problem. The two sources, in this case visual and spoken, have potentially very different interpretations or presumed reliability. When making decisions in real time, how do we combine cues coming from visual and linguistic evidence sources? In a sequence of three studies we asked participants to navigate through a set of virtual mazes using a head-mounted virtual reality display. Each maze consisted of a series of T intersections, at each of which the subject was presented with a visual cue and a spoken cue, each separately indicating which direction to continue through the maze. However the two cues did not always agree, forcing the subject to make a decision about which cue to “trust.” Each type of cue had a certain level of reliability (probability of providing correct guidance), independent from the other cue. Subjects learned over the course of trials how much to follow each cue, but we found that they generally trusted spoken cues more than visual ones, notwithstanding the objectively matched reliability levels. Finally, we show how subjects' tendency to favor the spoken cue can be modeled as a Bayesian prior favoring trusting such sources more than visual ones.