The McGurk Effect: Is Seeing Believing? Multimodal Sensory Perception at Work!

Although I have not talked about it much up until this point, if you have read my bio you might know that I am in fact a psychology major! This quarter in college I am taking Introduction to Perception, where I am learning all of the cool neural mechanisms for why we perceive the world the ways we do!

Last week, we focused on auditory perceptions and illusions. I learned about this really cool illusion that we all experience without realizing, that is also impossible to overcome even when aware of it (you may know of some other illusions that are easier to overcome when you are actively aware of the illusion, but this is not one of them!).

This illusion is called the McGurk Effect! Before I explain how it works, check out this video below by BBC Two where Dr. Lawrence Rosenblum (UC Riverside) illustrates the McGurk Effect as it applies to speech perception.
[I'm having a bit of trouble getting the embedded video to properly start and stop when I want it to, so for now just skip to 00:32 to start and pause the video again at about 1:09]

Pretty cool, right? So how does this illusion actually work? If I were to say it in one sentence, the McGurk Effect is essentially an interplay of categorical and multimodal perception. I'll break those two concepts down for you in the following paragraphs.

In the image to the right, you will see a spectrogram of different phonemes (a perceptually distinct sound in a language). You can see just how similar the different phonemes within each set are to each other, with the only difference being the location of articulation (where the tongue cuts off airflow in the mouth).

To help illustrate this further if the image is a little confusing, take a moment to make those sounds in your own mouth and really feel where that articulation is coming from

The average person can differentiate between these phonemes pretty accurately. This ability is known as categorical perception, or the ability to differentiate stimuli categorically. Even so, our categorical perception isn't perfect. There are some articulations that aren't easily categorized, and may fall into an overlapping range where a sound could be either "bah" or "dah", but we just aren't sure.

Fear not, for we have this handy-dandy skill called multi-modal perception! Multi-modal perception essentially means we use multiple forms of perception to differentiate between a set of stimuli. In this instance, we will use our visual perception to help us figure out if that person was saying "bah" or "dah". We instinctually understand that the mouth makes different shapes when articulating specific sounds, and we use these visual cues to help us determine what sound is being made. This is how deaf individuals are able to lip read, because they can memorize what mouth movement corresponds to what specific phoneme they would otherwise be hearing. The ability to lipread shows just how strong this multimodal capability really is!

Now, back to the video and the McGurk Effect. The entire time, Dr. Rosenblum is making a "bah" sound, which we can easily distinguish with categorical perception alone. However, when he switches his mouth movement to the inward curling lip articulation, we recognize that movement as "fah", and our visual perception overrides the audio perception. Even when you are aware of this, you cannot overcome the McGurk Effect. You will continue to hear "fah", even while we know he is saying "bah". If you close your eyes while listening, your perception will switch back from "fah" to "bah".

It is easy to take for granted just how powerful our brains actually are, and all of the mechanisms that make it possible for us to make sense of the perceptual world we live in. Make sure to follow me for more cool psychology posts, and let me know if there are any topics you are interested in learning more about! I hope you enjoyed learning about the McGurk Effect as much as I did, and for good measure here is that video once more :)