Biased competition in semantic representation during natural visual search

Mo Shahdloo, Emin Çelik, Tolga ÇukurbioRxiv 2019 · DOI

TL;DR

This study investigates how our brains handle visual search for multiple objects simultaneously. Using fMRI, researchers found that when we attend to multiple items, our brain's representation of those items is a weighted combination of how it represents each item individually. This combination is biased by the brain's inherent selectivity for certain categories, particularly in visual areas.

What Problem Does This Paper Address?

In everyday life, we constantly need to find multiple things in our visual field – think of scanning a crowded room for your friends or looking for specific items in a grocery store. As the number of things we're looking for increases, our visual search becomes more difficult. The "biased competition hypothesis" (BC) offers an explanation: our brains have limited capacity, so when multiple items compete for representation, the competition is "biased" by the brain's existing preferences for certain types of information. However, it was unclear if this hypothesis held true for complex, natural scenes and for high-level semantic representations (like "humans" or "vehicles") rather than just simple features. This paper aims to bridge that gap by testing the BC hypothesis in the context of natural visual search for object categories.

Methods

The researchers used functional Magnetic Resonance Imaging (fMRI) to record brain activity in five participants as they watched natural movies. During the movie watching, participants performed three different tasks in separate runs:

  1. Attend to "humans": Search for human-related objects.
  2. Attend to "vehicles": Search for vehicle-related objects.
  3. Attend to "humans and vehicles": Search for both simultaneously (divided attention).

They developed a sophisticated model to measure how the brain represented hundreds of object and action categories within these movies. This involved:

  • Category Modeling: Fitting models to voxelwise BOLD responses to estimate how each voxel responded to 831 different object and action categories.
  • Semantic Space Construction: Using Principal Component Analysis (PCA) on the response profiles from the single-target attention tasks to create a "semantic space" that captures the underlying relationships between categories.
  • Linearity Index (LI): To test if the representation during divided attention was a linear combination of the single-target representations, they predicted the divided attention tuning using a weighted sum of the single-target tunings. The LI was the correlation between the predicted and actual tuning.
  • Bias Index (BI): To assess if the divided attention representation was biased towards one target over the other, they regressed the divided attention tuning onto the single-target tunings and quantified the bias using regression weights.

Datasets

The study used 72 minutes of continuous natural movies as stimuli. These movies were compiled from various sources and were labeled for the presence of 831 distinct object and action categories. No new datasets were introduced; existing movie clips and a WordNet lexicon for category inference were utilized.

Key Insights and Ideas

  • Semantic Representation is Linear (Mostly): The brain's representation of semantic categories during divided attention isn't a complete mess; it's largely predictable as a weighted sum of how it represents each category individually. This suggests a structured, rather than chaotic, response.
  • Category Selectivity Matters: Brain areas that are highly specialized for certain categories (like faces or places) show a stronger "bias" in their representations during divided attention, favoring the category they are most sensitive to.
  • Attention Warps the Semantic Space: The way we attend to specific categories actively shapes how the brain represents them, even influencing the representation of non-target items that are semantically similar.
  • Beyond Individual Objects: The biased competition isn't just about competing for simple features; it extends to high-level semantic representations, showing how our brains integrate complex information.

Key Results

  • High Linearity: The linearity index (LI) was substantial across various brain regions, indicating that semantic tuning during divided attention can be well-explained by a weighted linear combination of single-target attention. For target categories, LI was around 0.81-0.88 across different functional areas.
  • Category-Selective Bias: In category-selective areas (like FFA for faces, PPA for scenes), the representation during divided attention was significantly biased towards the preferred target category. For example, in human-selective areas, representations were biased towards "humans."
  • Non-Target Bias: Even non-target categories that were semantically similar to the targets showed a biased representation, suggesting that attention influences not just what we're looking for, but also related concepts.
  • Area-Specific Differences: General object-selective areas (LOC) and attentional-control areas (IPS, FEF, FO) showed higher linearity than highly category-selective areas, suggesting these areas might have more flexible representations.

Limitations and Future Work

  • Natural Stimuli Correlations: The authors acknowledge that natural stimuli inherently contain correlations between features and categories. While they used a motion-energy regressor to mitigate this, residual correlations might still influence the results. Future work could use more controlled stimuli to minimize these correlations.
  • Linearity Assumption: The study primarily focused on linear models of competition. However, nonlinear models have been proposed for neural responses. Further research is needed to investigate potential nonlinear models of competition in semantic representations.
  • Individual Differences: While the study used five subjects, future research could explore individual differences in how biased competition manifests.

Summary updated

Abstract

Abstract Humans divide their attention among multiple visual targets in daily life, and visual search gets more difficult as the number of targets increases. The biased competition hypothesis (BC) has been put forth as an explanation for this phenomenon. BC suggests that brain responses during divided attention are a weighted linear combination of the responses during search for each target individually. Furthermore, this combination is biased by the intrinsic selectivity of cortical regions. Yet, it is unknown whether attentional modulations of semantic representations of cluttered and dynamic natural scenes are consistent with this hypothesis. Here, we investigated whether BC accounts for semantic representation during natural category-based visual search. Human subjects viewed natural movies, and their whole-brain BOLD responses were recorded while they attended to “humans”, “vehicles” (i.e. single-target attention tasks), or “both humans and vehicles” (i.e. divided attention) in separate runs. We computed a voxelwise linearity index to assess whether semantic representation during divided attention can be modeled as a weighted combination of representations during the two single-target attention tasks. We then examined the bias in weights of this linear combination across cortical ROIs. We find that semantic representations during divided attention are linear to a substantial degree, and that they are biased toward the preferred target in category-selective areas across ventral temporal cortex. Taken together, these results suggest that the biased competition hypothesis is a compelling account for attentional modulations of semantic representation across cortex. Significance Statement Natural vision is a complex task that involves splitting attention between multiple search targets. According to the biased competition hypothesis (BC), limited representational capacity of the cortex inevitably leads to a competition among representation of these targets and the competition is biased by intrinsic selectivity of cortical areas. Here we examined BC for semantic representation of hundreds of object and action categories in natural movies. We observed that: 1) semantic representation during simultaneous attention to two object categories is a weighted linear combination of representations during attention to each of them alone, and 2) the linear combination is biased toward semantic representation of the preferred object category in strongly category-selective areas. These findings suggest BC as a compelling account for attentional modulations of semantic representation across cortex in natural vision.
BibTeX
@misc{shahdloo2019biased,
  author = {Mo Shahdloo and Emin Çelik and Tolga Çukur},
  title = {{Biased competition in semantic representation during natural visual search}},
  year = {2019},
  doi = {10.1101/658096},
  howpublished = {bioRxiv},
}
Figure