Gaze-Modulated Feature Masking in Neural Networks

16 concepts13 papers1 proposal

Deep neural networks excel at pattern recognition but lack a fundamental human capability: selective attention. When a standard convolutional network or transformer processes a high-resolution image or video, it evaluates all pixels through its parametric filters. This often leads to information overload, where the model overfits to background noise or relies on spurious correlations.

Humans, by contrast, use eye movements to dynamically filter out irrelevant sensory data, focusing cognitive resources only on task-relevant regions. A prominent research direction bridges this gap by explicitly injecting human gaze data—or synthetic equivalents—directly into the intermediate feature representations of neural networks. By masking and re-weighting feature maps based on spatial attention, these systems mimic human selective attention to achieve more robust and sample-efficient learning.

The Core Mechanism: Element-Wise Feature Modulation

How do we mathematically force a network to "look" where a human looks? The most common approach is spatial feature re-weighting via element-wise multiplication.

Consider an intermediate convolutional feature map $F \in \mathbb{R}^{C \times H \times W}$, where $C$ is the number of channels, and $H \times W$ represents the spatial dimensions. A gaze-tracking device or an attention model generates a two-dimensional spatial heatmap or Gaze Density Map $M \in \mathbb{R}^{H \times W}$, where higher values correspond to regions of intense human focus. By normalizing $M$ and multiplying it element-wise across all channels of $F$, the network dynamically scales its activations:

$$F_{\text{modulated}} = F \odot M$$

Features within the human's line of sight are amplified, while background noise is suppressed to zero.

[Input Image/Video] ---> [Feature Extractor] ---> Feature Maps (F) 
                                                        |
                                                        v  (Element-wise Multiplication)
[Gaze/Interaction]  ---> [Spatial Heatmap]  ---> Attention Mask (M)
                                                        |
                                                        v
                                             [Modulated Features] ---> [Downstream Task]

This simple geometric gating has powerful implications across diverse tasks. For instance, Bulling et al. introduced a "Gaze Pooling Layer" that applies this exact mechanism to infer user search intent (such as identifying object categories or attributes) without requiring explicit verbal feedback.

In robotic vision, Falagario et al. fused normalized attention heatmaps with visual feature maps to resolve ambiguities in verbal requests. If a user asks a robot to "pass the box" in a room filled with boxes, multiplying the feature maps by the user's gaze mask effectively filters out distractor objects outside the line of sight. Uppal et al. demonstrated that this direct weighting strategy serves as an effective foundation for decoding human intent from complex, real-world scenes.

Predicting Attention When Gaze is Unavailable

While hardware eye-trackers provide ground-truth gaze maps during dataset collection, requiring a physical eye-tracker at inference time is often impractical. To circumvent this, researchers utilize a two-stage paradigm: first, train a specialized module to predict where a human would look, and second, use that predicted map to mask downstream features.

Ruohan Zhang demonstrated the power of this approach in reinforcement learning. By training a network to predict human expert gaze from raw Atari game frames, Zhang used the predicted attention map as a spatial mask to compress the state space, forcing the downstream policy network to focus only on critical game elements.

In video understanding, Yu et al. developed the Gaze Encoding Attention Network (GEAN), which applies predicted gaze weights to high-resolution visual and motion features, ensuring video captioning models focus on narrative-heavy regions.

Furthermore, human attention is not static; it shifts dynamically across moving objects. Fan et al. tackled this "saliency shift" by training models directly on dynamic human fixation maps rather than static binary object masks. This allows the system to learn the temporal transitions of human visual intent in complex, multi-object videos.

Why It Works: Inductive Biases and Noise Suppression

Standard neural networks must learn what is relevant purely from task labels (such as class names or rewards), which requires massive amounts of data and often leads to overfitting on background context. Gaze-modulated masking acts as a powerful inductive bias that constrains the network's search space.

As reviewed by Saran et al., integrating gaze data forces networks to manage information overload by mimicking human selective attention. In fine-grained classification tasks—such as distinguishing bird species or diagnosing medical X-rays—Xu et al. showed that using gaze data as a spatial mask guides the network toward compact, expert-aligned feature sets. Instead of learning to identify a bird based on the branch it sits on, the network is forced to process only the anatomical regions prioritized by human experts.

This concept even extends to cognitive modeling; Roseboom et al. demonstrated that filtering perceptual inputs with spatial attention data allows a classification network to accumulate salient environmental changes in a way that closely mirrors human subjective time perception.

Generalizing Beyond Gaze: Interactive and Spatial Gating

The principle of spatial feature re-weighting is not restricted to eye-tracking; it generalizes to any external spatial signal, such as user clicks, geometric configurations, or cross-view consensus.

In interactive remote sensing, Hu et al. designed an attention module that dynamically re-weights multi-scale features based on user input to segment complex land covers like forests and water bodies. Similarly, the authors of IRTDETR leveraged attention maps derived from user interactions to focus object detection models on tiny objects in massive satellite images, drastically reducing annotation effort.

Beyond explicit user input, spatial relationships can inherently define attention masks. For example, the authors of VSGNet convert the spatial configurations of human-object pairs into binary location maps, using them as attention weights to amplify features in interacting regions while suppressing background noise.

Finally, this spatial anchoring can bridge domain gaps. Qiu et al. used predicted gaze points as spatial anchors to filter out task-irrelevant backgrounds, creating a shared representational space between first-person (egocentric) and third-person (exocentric) video perspectives. Because both the actor and the observer look at the same task-relevant objects, gaze-modulated filtering provides a invariant bridge across different camera viewpoints.

Key Foundational Work

Predicting the Category and Attributes of Visual Search Targets Using Deep Gaze Pooling (2017) — contributed 1 concept elements to this direction.
Supervising Neural Attention Models for Video Captioning by Human Gaze Data (2017) — contributed 1 concept elements to this direction.
Activity in perceptual classification networks as a basis for human subjective time perception (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_attention_guided_feature_re_weighting_9f35b2ed,
  title = {attention-guided feature re-weighting},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/attention-guided-feature-re-weighting/rdr_9f35b2edbe5a4129a32759b45d8cda3d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_attention_guided_feature_re_weighting_9f35b2ed_v1,
  title = {attention-guided feature re-weighting},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/attention-guided-feature-re-weighting/rdr_9f35b2edbe5a4129a32759b45d8cda3d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake