Traditional computer vision models often treat visual attention as a "bottom-up" process. In this view, our eyes are drawn to pixels with high contrast, bright colors, or sharp edges. However, human behavior tells a different story: we don't just look at pixels; we look at things. Whether it is a face in a crowd, a stop sign while driving, or a headline on a webpage, our gaze is governed by semantic meaning.
Recent research has shifted toward Object-Centric Gaze Prediction, a paradigm that integrates high-level semantic understanding with low-level visual features to better replicate how humans actually perceive the world.
The Failure of Pure Saliency
Early saliency models focused on "feature maps"—mathematical representations of intensity, color, and orientation. While these models can predict where a person might look in an abstract pattern, they struggle with natural scenes. As Rui Zhao and Xiaogang Wang argue in Saliency detection by multi-context deep learning, saliency is more effectively treated as a high-level semantic task. A red apple is salient not just because it is red, but because it is an "apple" sitting on a "table."
When models ignore these categories, they often fail to detect objects that share similar visual characteristics with the background. By prioritizing object category recognition, AI can identify salient regions even in low-contrast environments.
The Power of Semantic Channels
One of the most potent attractors of human attention is the human face. Moran Cerf and Christof Koch demonstrated in Predicting human gaze using low-level saliency combined with face detection that observers fixate on a face within their first two eye movements nearly 90% of the time.
By adding a dedicated "face channel"—essentially a specialized object detector—into a standard saliency model, they significantly improved gaze prediction. This proved that certain high-level semantic categories function as primary attractors of attention, operating in parallel with low-level features like color and intensity.
Integrating External Knowledge
Humans don't look at scenes with a blank slate; we bring a lifetime of "priors" or external knowledge. We know that a toaster is likely to be near a kettle, and a pedestrian is likely to be on a sidewalk.
To mimic this, researchers like Yifeng Zhang and Qi Zhao developed GraSSNet (Saliency Prediction with External Knowledge). This model explicitly incorporates taxonomic hierarchies (from WordNet) and object co-occurrence data. By modeling the relationships between objects as a graph, the system can predict attention based on how humans naturally understand scene context, rather than just learning from raw pixel data.
Task-Specific and Personalized Attention
The "where" of human gaze is often dictated by the "why." In Predicting the Driver's Focus of Attention: The DR(eye)VE Project, Andrea Palazzi and Davide Abati show that driving attention is governed by semantic relevance—the road, other vehicles, and pedestrians—rather than brightness or contrast.
Furthermore, attention is subjective. Sikun Lin and Pan Hui 0001 introduced PANet (Where's YOUR focus: Personalized Attention), which uses a dual-stream architecture. One stream handles general visual prominence, while the second weights detected objects based on a user’s specific interest profile (e.g., a person interested in cars will look at the vehicle, while a botanist might look at the trees).
From Heatmaps to Scanpaths
Most early models produced static "heatmaps," but human vision is dynamic. We move our eyes in a sequence called a scanpath. Ali Borji and Laurent Itti found that these transitions are better predicted by modeling movements between semantic objects rather than pixel-to-pixel jumps (An Object-Based Bayesian Framework for Top-Down Visual Attention).
This object-level approach is refined in the Object-level Attention Transformer (OAT) by Yini Fang and colleagues (OAT: Object-Level Attention Transformer for Gaze Scanpath Prediction). By treating an image as a collection of discrete entities rather than a grid of patches, OAT mimics human foveated vision, prioritizing objects based on their semantic importance and spatial relationships.
Why It Works: The Semantic Bridge
The core intuition behind this research direction is that the human brain's attention mechanism operates on an object-level representation to fulfill specific goals. By bridging the "semantic gap"—the distance between raw sensory input and conceptual meaning—these models achieve several things:
- Robustness: They ignore "distractor" pixels that have high contrast but no meaning.
- Efficiency: They use pre-existing knowledge to narrow down search areas.
- Explainability: They allow us to understand why a model (or a human) looked at a specific spot, such as "the pedestrian" rather than "the bright red wall."
As we move toward autonomous systems and personalized AI, the ability to predict not just where a human looks, but the meaning behind that gaze, becomes the foundation for truly human-centric technology.