Abstract
Efficient multisensory integration is often influenced by other cognitive processes including, but not limited to, semantic congruency and focused endogenous attention. Semantic congruency can re-allocate processing resources to the location of a congruent stimulus, while attention can prioritize the integration of multi-sensory stimuli under focus. Here, we explore the robustness of this phenomenon in the context of three stimuli, two of which are in the focus of endogenous attention. Participants completed an endogenous attention task with a stimulus compound consisting of 3 different objects: (1) a visual object (V) in the foreground, (2) an auditory object (A), and (3) a visual background scene object (B). Three groups of participants focused their attention on either the visual object and auditory sound (Group VA, n = 30), the visual object and the background (VB, n = 27), or the auditory sound and the background (AB, n = 30), and judged the semantic congruency of the objects under focus. Congruency varied systematically across all 3 stimuli: All stimuli could be semantically incongruent or all could be congruent, or two objects could be congruent with the remaining one incongruent to the other two. Participants exhibited a distinct pattern of errors: when participants attended two congruent objects in the presence of an unattended, incongruent third object, they tended to make more errors than in any other stimulus combination. Drift diffusion modeling of the behavioral data revealed a significantly smaller drift rate in the two-congruent-attended condition, indicating slower evidence accumulation, which was likely due to interference from the unattended, incongruent object. These findings implicate that our perceptual system is highly sensitive to semantic incongruencies even when they are not endogenously attended.
Introduction
In our multisensory world, our brains constantly integrate information from different senses. Two major factors drive this: semantic congruency (do these things belong together, like a "meow" and a cat?) and endogenous attention (what we choose to focus on).
Previous research offered conflicting views. One line of thought suggests that semantic congruency acts as a "top-down" booster, helping us focus better. Another suggests that incongruency acts as a "bottom-up" distractor, forcing the brain to divert resources to resolve the conflict. This paper investigates the robustness of these effects by asking: What happens when we are focused on two matching items, but a third, irrelevant item doesn't fit the scene? Does our voluntary attention shield us from this "semantic noise," or does the mismatch "leak" in and interfere?
Methods
The researchers recruited 87 participants and split them into three groups. Each participant was presented with a "stimulus compound" consisting of three elements:
- Visual Foreground (V): An object (e.g., a duck).
- Auditory (A): A sound (e.g., a quack).
- Visual Background (B): A scene (e.g., a phone booth).
Each group was told to ignore one element and judge the "plausibility" (semantic congruency) of the other two. For example, Group VA focused on the object and the sound.
The researchers used Drift Diffusion Modeling (DDM) to analyze the behavioral data. Unlike simple accuracy or reaction time metrics, DDM decomposes the decision-making process into latent variables, such as the "drift rate" (the speed of evidence accumulation) and "boundary separation" (how much evidence is needed before making a choice). This allowed them to see how the distraction was affecting the participants' brains.
Datasets
The study utilized a custom-built set of stimuli:
- Visual objects: 250-pixel high images of animals and everyday items.
- Auditory objects: Typical sounds associated with those visual objects (mean duration 1.53s).
- Background scenes: 768 x 1024 pixel images depicting typical contexts (e.g., a savannah, a kitchen).
Key Insights and Results
The most striking finding was that participants made significantly more errors when the two items they were attending to were congruent, but the ignored item was incongruent. If you are looking at a lion and hearing a roar, your brain "expects" a savannah. If it sees a bathroom instead, it trips up—even if you're trying to ignore the background.
- Modality Independence: This interference happened regardless of which senses were being used. Whether participants were focusing on Visual-Auditory, Visual-Background, or Auditory-Background pairs, the "unattended mismatch" effect remained consistent.
- Slower Evidence Accumulation: The DDM analysis revealed a significantly lower drift rate in these critical conditions. This suggests that the incongruent distractor isn't just a random distraction; it actively interferes with the brain's ability to gather and process evidence for the task at hand.
- Automatic Conflict Detection: The results imply that our perceptual system monitors for semantic consistency across the entire scene automatically. We cannot "shut off" the processing of background information if it contains a glaring semantic conflict.
- Error Rates: While overall error rates were low (8-13%), they spiked dramatically in the "critical" conditions (e.g., attending to V and A while B was incongruent).
- Response Bias Ruled Out: A control analysis confirmed that the errors weren't just due to a "negative bias" (a tendency to say "no" when something felt off); rather, it was a genuine failure to correctly integrate the attended stimuli in the face of conflict.
Discussion
While errors increased, reaction times (RT) didn't show a perfectly corresponding increase on error trials. The authors suggest this might be because once the brain "re-focuses" on the distractor, it makes an "incongruent" judgment quite quickly, even if that judgment is technically "incorrect" for the specific task.
While the paper discusses the likely involvement of the Anterior Cingulate Cortex (ACC) based on previous literature, this study was purely behavioral. Future work combining this paradigm with fMRI or EEG could pinpoint exactly when and where this "semantic leak" occurs in the processing stream. Additionally, the stimuli were relatively simple (one object, one sound, one background). Real-world environments are much noisier, and it remains to be seen how these effects scale in more complex, naturalistic settings.
Summary updated