Audiovisual interactions in time and spatial grouping principles of vision

Abstract

Multisensory integration is often studied with intermodal con ict where either visual input dominates and alters the percept of simultaneous auditory input or the other way around. For instance, when put in con ict, visual stimuli can drive the perception of where a sound originates (spatial ventriloquism) [1, 2] whereas auditory stimuli can drive the perception of when visual events occur (temporal ventriloquism) [3, 4, 5] .These interactions make adaptive sense given the auditory system´s superior temporal resolution and the visual system´s superior spatial resolution [6]. Moreover, it was found that temporal ventriloquism can change the perceived speed of visual motion [7]. By taking advantage of this in uence of auditory timing on perceived speed, we investigated how audiovisual interactions in time (i.e., temporal ventriloquism) are modulated by the spatial grouping principles of vision. In our experiments, we manipulated spatial proximity, common fate and uniform connectedness between moving ashes. Observers compared the speed of motion between different auditory timing conditions. Our results revealed that auditory in uences on perceived speed were significantly modulated by only uniform connectedness. More specifically, we found that auditory effects on vision was significantly less when a horizontal gray connecting bar grouped multiple sequential moving ashes. When horizontally placed moving ashes were grouped with a vertical connecting bar, the degree of auditory in uences in time was significantly stronger compared to not grouped (control) and horizontal connecting bar conditions. The effect of auditory clicks on single apparent motion grouped with horizontal connecting bar was smaller relative to the not-grouped condition. In addition, our analysis on EEG activities revealed consistent trends in agreement with the behavioral results. Audiovisual interaction patterns were observed both earlier (around P1) and later (around N1 and P2). Less auditory capture over the horizontal connecting bar condition relative to the vertical connecting bar condition was observed around 50-100 ms (P1) on the frontal and temporal channels and around 200-300 ms (N1) on the frontal, central, temporal and occipital-parietal channels. The larger effect of sound over the single apparent motion without connecting bar condition was observed around 50-100 ms and 200-300 ms on the central channels and 200-300 ms on the occipital-parietal channels. The difference between the individual effects of inner and outer sound conditions was found to be less for the horizontal connecting bar condition relative to the vertical connecting bar condition over the frontal, temporal, central and occipital-parietal channels while less individual effects of the inner and outer sound conditions on the single apparent motion with horizontal connecting bar condition relative to the without connecting bar condition was only observed over the central channels and late time intervals of the occipital-parietal and temporal channels. Overall the individual effects of both sound conditions were consistently similar on the horizontal connecting bar condition (compared to the vertical connecting bar condition) for all ROIs and on the single apparent motion with horizontal connecting bar condition (compared to the no-connecting bar condition) over the central channels. In general, our findings here suggest that temporal ventriloquism effects exist in different spatial grouping conditions of vision, but they can be also modulated by certain intra-modal grouping principles such as uniform connectedness.