When you look at a busy street scene, your eyes are instantly drawn to a pedestrian stepping off the curb, a stop sign, or text written on a billboard. You do not spend time staring at a highly textured brick wall or a blank patch of bright asphalt, even if those regions take up the majority of your visual field.
For decades, computer vision models struggled to replicate this behavior. Early computational models treated visual saliency—the problem of predicting what parts of an image catch human attention—as a purely mathematical or statistical exercise. They looked for local contrast, sharp edges, or rapid changes in color.
This disconnect creates what researchers call the semantic saliency gap: the profound mismatch between low-level visual features (what a pixel looks like) and high-level semantic meaning (what an object represents to a human observer). Bridging this gap is the central challenge of modern visual attention modeling.
Why Purely Bottom-Up Models Fail
Early breakthroughs in saliency modeling were fundamentally "bottom-up." Models like the one proposed by Jonathan Harel et al. in Graph-Based Visual Saliency treated images as graphs and defined saliency as an equilibrium state of a Markov chain based on local feature unusualness. Similarly, Neil D. B. Bruce et al. developed a model based on Saliency Based on Information Maximization, which calculated the "self-information" of local image patches to find statistically rare regions.
While these models are mathematically elegant and biologically plausible for the earliest stages of human visual processing (the primary visual cortex), they break down in complex, real-world environments.
The fundamental flaw of a purely bottom-up approach is that statistical rarity does not equal semantic importance. As benchmarks scaled up to capture naturalistic scenes, the limitations became glaring. The creators of the SALICON: Saliency in Context dataset demonstrated a massive performance gap between computational models and actual human observers on non-iconic, complex images. A bottom-up model might flag a flickering light or a high-contrast geometric pattern as highly salient, completely missing the subtle social cues, text, or narrative elements that naturally anchor human gaze.
The First Bridge: Explicit Semantic Channels
To fix this, researchers realized they needed to inject "top-down" cognitive priors directly into the models. The most intuitive way to do this was to build dedicated detectors for things humans are biologically hardwired to look at.
Faces are the ultimate example of this biological bias. Moran Cerf et al. demonstrated this effect in Predicting human gaze using low-level saliency combined with face detection. By introducing an explicit "face channel" alongside traditional low-level feature maps, the authors significantly boosted gaze prediction accuracy. They proved that certain high-level semantic categories are so fundamental to human cognition that they override raw pixel contrast entirely; an observer will fixate on a face within their first two eye movements, regardless of how well that face blends into the background color-wise.
Shifting from Pixels to Objects and Context
As deep learning matured, the strategy shifted from hand-crafting specific category detectors (like faces or text) to fundamentally changing how models perceive a scene. Humans do not look at isolated spatial coordinates; we look at discrete, bounded objects to fulfill goals or understand narratives.
Ali Borji et al. formalized this in An Object-Based Bayesian Framework for Top-Down Visual Attention, moving away from pixel-level maps toward Dynamic Bayesian Networks that track objects over time. This object-centric view is crucial because human attention is deeply sensitive to context and semantic violations.
For instance, Marek A. Pedziwiatr explored how object-scene inconsistencies alter human eye movements in Semantic object-scene inconsistencies affect eye movements, but not in the way predicted by contextualized meaning maps. If a completely unexpected object appears in a scene—like a flip-flop sitting on a kitchen counter—our attention is immediately drawn to it because it violates our top-down expectations of the world. Standard feature-driven maps are entirely blind to these contextual anomalies because, structurally, the flip-flop might match the background perfectly.
Closing the Loop with Language Guidance
The modern frontier of this research direction completely bypasses the need to manually define semantic rules by leveraging multimodal language-vision models. Language provides a dense, structured prior for what makes a scene meaningful.
Instead of relying solely on image features, models can now use textual descriptions to guide visual focus. Jonghwan Mun et al. pioneered this in their Text-Guided Attention Model for Image Captioning, using retrieved captions from similar images to "steer" the model's attention toward contextually relevant regions before generating new text.
Taking this a step further, Dario Zanca et al. proved in Contrastive Language-Image Pretrained Models are Zero-Shot Human Scanpath Predictors that models like CLIP can predict chronological human eye-movement sequences (scanpaths) without ever being trained on actual eye-tracking data. By aligning text prompts with visual features, these models simulate how language and high-level concepts naturally dictate the sequence of human fixations.
By evolving from low-level pixel statistics to explicit semantic channels, object-level context, and finally multimodal language grounding, computer vision has successfully closed the semantic saliency gap. Saliency is no longer viewed as just a property of contrast, but as a reflection of human meaning.