A  person perceives the world around them through multiple senses, with concurrent impulses of signals in different sensory modalities, such as a flash and a beep, readily deciphered and merged into a single event. However, real-world dynamic events often generate signals with different onsets and durations across the brain and the senses, raising the possibility of 4-D perception.  It is hoped mapping these areas in the brain will enable the development of 4-D or multisensory electronic systems.  Now, a study led by researchers at UC San Diego develops a pair of 4-D glasses allowing wearers to be physically ‘touched’ by a movie when they see a looming object on the screen, such as an approaching spacecraft.  The team states their device was developed in conjunction with a study to map brain areas integrating the sight and touch of a looming object to aid in their understanding of the perceptual and neural mechanisms involved in multisensory integration.  The opensource study is published in the journal Human Brain Mapping.

Previous studies indicate real-world objects approaching or passing by an observer often generate visual, auditory, and tactile signals with different onsets and durations. Prompt detection and avoidance of an impending threat depend on the precise binding of looming signals across the brain and senses.  The current study uses multisensory apparatus integrating a direct-view wide-field screen with flexible air hoses to deliver spatially aligned looming visual and tactile stimuli near the face with a varying temporal offset.

The current study assesses the synchrony between a looming ball simulated in virtual reality and an air puff delivered to the same side of the face. Results show the air puff was perceived as completely out of sync with the looming ball when the onset of ball movement and the onset of an air puff were nearly simultaneous.  Data findings show the two stimuli were perceived as one when there was a delay between 800 to 1,000 milliseconds between them as if an object had passed near the face generating a little wind.

In experiments using functional Magnetic Resonance Imaging (fMRI), the group delivered tactile-only, visual-only, tactile-visual out-of-sync, and tactile-visual in-sync stimuli to either side of the subject’s face in randomized events. Results show more than a dozen brain areas were found to respond more strongly to lateralized multisensory stimuli than to lateralized unisensory stimuli, and the response was further enhanced when the multisensory stimuli are in perceptual sync.

The team surmises their study demonstrates the perceptual and neural mechanisms of multisensory integration near the face, with potential applications for the development of 4-D multisensory entertainment systems and media.  For the future, the researchers state their studies will help to further understand the perceptual and neural mechanisms of multisensory integration, and provide a solid scientific foundation for developing 4-D film.

Source: UC San Diego

Get Healthinnovations delivered to your inbox: