Visual and auditory information is essential for understanding speech. When we talk to someone, we not only hear what they are saying, but we also watch how they say it by moving their lips.
The ability of human speech is based on the integration of visual and auditory information, which is evidenced by the fact that we can experience a curious illusory phenomenon: the McGurk effect.
It can be said that this particular phenomenon occurs when we feel with our eyes, causing what we feel to change according to what we see. We find out what this interesting visual-auditory effect is all about.
What is the McGurk effect?
We tend to believe that our senses work independently: when we feel, we are only feeling; and when we see, we only see. Based on this belief, it would be reasonable to think that a visual stimulus is not capable of distorting the way we perceive sound. However, the reality is that it can, as our perceptual experiences are the product of a complex process of mixing information, the same mixing that gives rise to a particular phenomenon: the McGurk effect.
Surely you have had a conversation more than once in an extremely noisy environment. Maybe it was in a nightclub, on the terrace of a busy street bar, or in a high school classroom. When there is a loud background noise, it is difficult for us to understand what the person in front of us is saying to us, and to understand something we use the old instinctive trick of watching the mouth while speaking.
In these cases, visual and auditory information is not analyzed separately, but combined. The human brain has a region called the superior temporal sulcus, which specializes in combining the two types of information, in the examples we have given would be responsible for combining the phonemes spoken by our interlocutor with the movement of the lips.
Due to this ability to combine multimodal information, the superior temporal sulcus is the neurological stage where the McGurk effect occurs, which would be nothing more than the result of a message decoding error when two different sensory modalities interact, causing what we see does not match what we hear.
If we do a quick search on Youtube, we can find more than one video where this phenomenon is practically exhibited. This link brings us to a good example of this phenomenon:
In this particular case, the person in the video is pronouncing / ba / all the time, but depending on how you move your lips, you may feel either / ba / or / pa /.
This effect can also be found with other combinations of syllables. For example, this can be achieved with the combination / ka / (visual) and / pa / (auditory), which results in the perception of / ta /. Another example would be seeing someone making lip movements that match the syllable / ga / but while the syllable / ba / is being spoken, it will be perceived as / da /.
The way we hear the same sound varies greatly depending on whether or not we watch the way the person speaking to us moves their lips. Not only does this affect the perception of simple sets of sounds such as syllables, it has also been shown to work with full sentences, although you have probably witnessed this yourself in some of the situations we have. previously mentioned.
One of the first discoveries related to the McGurk effect and the interplay between sensory modalities is to have the possibility of seeing the lips of our interlocutor move considerably improves the volume of what we feel.
We have seen that this gives us the impression that we can hear phonemes up to 15 decibels louder when we have the transmitter in our visual field. This happens even when the acoustic conditions are not unfavorable, such as being in a room without sound or in a quiet place.
History of its discovery
This phenomenon was first described in 1976 in an article by British cognitive psychologist Harry McGurk with his colleague John MacDonald titled “Hearing Lips and Seeing Voices”. His original study consisted of studying the imitation patterns of a group of children who were developing speech skills and the experiment involved presenting several videos of people speaking different syllables.
However, a read error has occurred. The technician in charge of editing the video made a mistake and caused the picture and sound to be out of sync, revealing the recording of a person saying something that did not match the sound heard.
As the video played, McGurk and MacDonald heard a third phoneme instead of what was being articulated with the lips and what was being emitted. It was a happy coincidence that these two researchers discovered this particular auditory illusion.
Its importance in the study of human speech
The discovery of this effect is considered a test that the visual and auditory system have evolved together to allow, among other things, better speech processing. Our visual system helps us discriminate between sounds that are difficult to differentiate, an advantage that people who are deaf enjoy when they read their lips.
Being able to see how our interlocutor moves their lips increases confidence in the message perceived through the auditory system. That is, if two independent systems point to the same solution, in this case the same message, we trust that message more than if we only receive it through one channel.
It should be mentioned that the McGurk effect does not happen automatically. In order for this to happen, we need to pay attention to our interlocutor, and when distracting stimuli, both visual and auditory, are incorporated, this illusion is lessened.
In fact, this proves that the effect is not due to poor reception of visual or auditory information, but to an error in the integration of these two sensory modalities.
Another fact that gives strength to the idea that the visual system supports hearing is that when we see a person speaking to us but cannot hear what they are saying to us at all, not only our visual cortex is activated, but also our auditory cortex, even if we don’t listen to anything.
The McGurk effect and brain dysfunctions
We saw that brain damage and impaired reading skills, in addition to manifesting mental disorders, influences the likelihood of the McGurk effect occurring.
People who have had a callosotomy show the McGurk effect more slowly. It appears that children with a specific language disorder exhibit the McGurk effect more weakly compared to children without language acquisition disorders or literacy difficulties.
Laterality also influences, as skillful people are more likely to experience this effect.
The McGurk effect in different languages
Regardless of the language spoken, its speakers depend to some extent on visual information when perceiving speech. However, we have seen that the intensity of the McGurk effect varies from one language to another, since in languages such as Spanish, Turkish, Italian, Dutch, English and German, its speakers feel this effect more strongly than Chinese and Japanese speakers.
The fact that speakers of Asian languages exhibit the McGurk effect less frequently may be due to the cultural practice of avoiding eye contact.. Added to this, Chinese and Japanese in particular are two languages with very syllabic linguistic structures, usually of consonant + vowel and consonant + vowel + consonant type, which makes them particularly good at detecting syllables on the fringes of the way their sound is. interlocutor moves his lips.
- McGurk, H., MacDonald, J. (1976). Listen to the lips and see the voices. Nature 264, 746-748. https://doi.org/10.1038/264746a0
- Prera, A (2021, June 10). The McGurk effect. Simply psychology. https://www.simplypsychology.org/mcGurk effect.html
- Bovo, R., Ciorba, A., Prosser, S. and Martini, A. (2009). The McGurk phenomenon among Italian listeners. Acta Otorhinolaryngologica Italica, 29 (4), 203.
- Sekiyama, K. (1997). Cultural and linguistic factors in audiovisual speech processing: the McGurk effect on Chinese participants. Perception and Psychophysics, 59 (1), 73-80.
- Sekiyama, K. and Burnham, D. (2008). Impact of language on the development of auditory-visual perception of speech. Developmental Science, 11 (2), 306-320.
- Erdener, D. (2015). McGurk’s Illusion in Turkish. Turkish Journal of Psychology. 30 (76): 19-31.
- Hisanaga, S., Sekiyama, K., Igasaki, T. and Murayama, N. (2009). Perception of audiovisual speech in Japanese and English: differences between languages examined by event potentials. A AVSP (p. 38-42).
- Schmid, G., Thielmann, A. and Ziegler, W. (2009). The influence of visual and auditory information on the perception of oral speech and non-vocal movements in patients with lesions of the left hemisphere. Clinical and Phonetic Linguistics, 23 (3), 208-221.
- random ev. (sf). McGurk effect (video). Youtube: https://www.youtube.com/watch?v=R_GXdTdbzjI