Sounds can change the way we see things

In our daily lives we are exposed to numerous objects that we need to recognize. For example, if something flies past the window at dusk, we would probably like to know whether this was a bird, a bat or a drone. Visual object recognition is seldom based on what our eyes perceive alone, but is influenced by the context and by our prior experiences. In addition, a new experimental study shows that what we see can also be modified by sounds that we hear at the same time, meaning that sounds can alter the way in which we perceive visual cues. 

The research study, published in Psychological Science, suggests that relevant audio cues can help us to recognize objects more quickly and accurately, but can also alter our visual perception. If we see a bird and hear birdsong, we quickly identify a bird. But if the presence of a bird is associated with the call of a squirrel, we become less sure of what we are looking at. The brain therefore makes use of audio input to help us decide what, exactly, we are seeing. 

“Your brain spends a significant amount of energy to process the sensory information in the world and to give you that feeling of a full and seamless perception,” said study lead author Jamal R. Williams from the UC San Diego. “One way that it does this is by making inferences about what sorts of information should be expected.” 

In the natural world, sounds are reliable predictors of the objects that cause them – dogs bark and cats meow. In this way, sounds provide independent and informative cues about the visual world. However, when we use these trusted audio cues during the early stages of identifying a visual object, they can lead us to selectively process only certain aspects of the visual features.

Williams explained that “informed guesses” based on audio input can help us to process information more quickly, but they can also lead us astray when what we’re hearing doesn’t match up with what we expect to see. 

The researchers conducted three experiments in which participants were shown ambiguous images of objects morphing into other objects. For example, a plane became a bird and the images shown to participants could come from anywhere along the process and have features of both a plane and a bird.

In the first experiment, 40 participants were presented with ambiguous figures and, at the same time as they were trying to discriminate the nature of the object, the researchers played either a sound related to the image (birdsong or the buzz of a plane), or an unrelated sound (such as a hammer hitting a nail). After exposure to the image, participants were asked to recall which stage of the object morph they had been shown. 

To identify the stage that they recalled, the participants used a sliding scale that could be moved in either direction, more towards a recognizable bird or more towards a recognizable plane. The results revealed that participants made their object-morph selection more quickly when they heard related (versus unrelated) sounds and that they shifted their object-morph selection to match more closely the related sounds that they had heard. The findings indicate that, when visual input is not clear-cut, the role of relevant sounds may modulate our visual perception.

“When sounds are related to pertinent visual features, those visual features are prioritized and processed more quickly compared to when sounds are unrelated to the visual features. So, if you heard the sound of a birdsong, anything bird-like is given prioritized access to visual perception,” Williams explained. “We found that this prioritization is not purely facilitatory and that your perception of the visual object is actually more bird-like than if you had heard the sound of an airplane flying overhead.” 

The researchers wondered at what stage of visual recognition the audio input was having its influence – was this during the stage of visual discrimination or during the time that the participants made their decision about the object’s identity. The second experiment presented 105 participants with similar ambiguous images but coincided the audio stimuli with either the object recognition stage (when the image was on the screen) or the decision-making stage (after the image was removed and the participant had to choose its identity).  

Once again, the audio input was found to influence participants’ speed and accuracy when the sounds were played while participants were viewing the object morph, but it had no effect when the sounds were played while they reported which object morph they had seen. The researchers concluded that real-world sounds primarily affect the perceptual process, and not the decisional process.

In the third and final experiment, the researchers played the sound to 40 participants before the object morphs were shown to them on screen. The aim of this was to test whether prior audio input influenced visual perception by affecting people’s expectations and priming them to pay more attention to certain features of the objects. This was also found to have no effect on participants’ object-morph selections. 

The experts conclude that, when taken together, these findings suggest sounds alter visual perception only when audio and visual input occur at the same time. 

“Our results suggest that naturalistic auditory information alters the representations of objects we see. Specifically, we found that visual features of object representations are shifted toward features that are congruent with a concurrent auditory stimulus: the same ambiguous object (e.g., a 50 percent seal and 50 percent hammer morph) was perceived as more hammer-like when paired with a hammer sound and more seal-like when paired with the sound of seal barking,” wrote the study authors.

“This process of recognizing objects in the world feels effortless and fast, but in reality it’s a very computationally intensive process,” said Williams. “To alleviate some of this burden, your brain is going to evaluate information from other senses.” 

The researchers would like to build on these findings by exploring how sounds may influence our ability to locate objects, how visual input may influence our perception of sounds, and whether audiovisual integration is an innate or learned process. 

—

By Alison Bosman, Earth.com Staff Writer

Check us out on EarthSnap, a free app brought to you by Eric Ralls and Earth.com.