Aligning Neural Representations with Auxiliary Gaze Prediction

24 concepts17 papers1 proposal

Deep neural networks excel at statistical pattern recognition, but they often struggle with selective attention—the ability to filter out background noise and focus strictly on task-relevant information. When processing a complex image or reading a dense paragraph, a standard model treats all input features with relatively uniform initial weight, relying on deep layers to eventually isolate what matters.

Humans, by contrast, navigate the world via an optimized biological search engine: our eyes. Where we look is not random; it directly reflects our intent, cognitive load, and semantic understanding. By forcing neural networks to predict where humans look as an auxiliary task, researchers can inject human cognitive priors directly into a network's latent space, significantly boosting performance on a wide array of primary objectives.

The Multi-Task Attention Synergy

To give a neural network a human-like filter, models are trained using a multi-task learning framework where gaze prediction serves as a secondary supervision signal. Instead of optimizing a model solely on a primary objective—such as action recognition, text comprehension, or medical diagnosis—the network is optimized simultaneously to predict human gaze metrics, such as saliency heatmaps, scanpaths, or 3D gaze vectors.

                  ┌───► Primary Task Loss (e.g., Action, QA)
┌─────────┐       │
│ Shared  ├───────┤
│ Encoder │       │
└────▲────┘       └───► Auxiliary Gaze Loss (e.g., Saliency Map)
     │
 Input Data

As noted in a comprehensive survey by Giuseppe Cartella et al., this joint optimization forces the model's shared encoder to develop internal representations that inherently align with human-like selective attention. The auxiliary loss acts as a regularizer, shaping the shared feature space so that the network naturally discards background clutter and prioritizes regions of high semantic importance.

Grounding Semantics and Filtering Noise

Why does this joint optimization improve the primary task? In visual environments, human gaze serves as a dynamic spatial index that grounds abstract concepts into concrete visual features.

Consider egocentric (first-person) video analysis. A person making coffee looks at the kettle, then the cup, then the spoon. In In the Eye of Beholder: Joint Learning of Gaze and Actions in First Person Video, Yin Li et al. demonstrate that treating gaze as a probabilistic latent variable within a joint action-gaze framework allows a model to ignore background clutter and focus on task-relevant objects. This creates a bidirectional synergy: knowing the action constrains where the person is likely looking, and predicting the gaze point clarifies fine-grained action boundaries.

Yifei Huang et al. formalized this bidirectional relationship in their Mutual Context Network (Mutual Context Network for Jointly Estimating Egocentric Gaze and Action), proving that action-dependent gaze patterns resolve visual ambiguities that single-task models fail to untangle.

This grounding effect extends beyond video to language-guided robotics. For instance, Winson Chen et al. showed that forcing drone navigation models to predict human visual attention masks alongside movement waypoints significantly improves how the model grounds natural language instructions into specific visual landmarks. The auxiliary gaze task forces the model to learn which visual features are relevant to the linguistic tokens in the command.

Encoding Cognitive Load and Information-Seeking Strategies

The utility of auxiliary gaze prediction is not limited to computer vision; it provides a powerful inductive bias for Natural Language Processing (NLP). When humans read, our eyes do not move linearly. We linger on complex words, skip predictable text, and regress (look backward) when a sentence structure is confusing.

As surveyed by Diptesh Kanojia et al., these eye-tracking attributes directly reflect human cognitive load and text processing strategies. By incorporating gaze prediction as an auxiliary objective, language models learn to mimic these reading patterns.

For example, in machine reading comprehension, Jonathan Malmaud et al. trained a transformer model to simultaneously answer questions and predict human fixation patterns. The auxiliary gaze task acts as an inductive bias that teaches the model information-seeking strategies—specifically, how to locate and linger on text segments that are highly relevant to the query.

Similarly, Sandeep Mathias et al. applied this to zero-shot automatic essay grading (Cognitively Aided Zero-Shot Automatic Essay Grading), demonstrating that predicting human reading behaviors allows the network to capture universal linguistic features that generalize to entirely unseen essay topics.

Overcoming Data Scarcity via Expert Logic

One of the most compelling reasons this paradigm works is its ability to regularize models when training data is scarce. In specialized domains like medical image analysis, standard deep networks require massive datasets to avoid overfitting to spurious correlations. However, human experts (like radiologists) possess highly optimized visual search patterns developed over years of training.

In Leveraging Human Selective Attention for Medical Image Analysis with Limited Training Data, Yifei Huang 0002 et al. utilized a joint loss function optimizing for both image segmentation and expert gaze estimation. This dual-objective training forces the latent representations of the network to capture the underlying diagnostic logic of the expert, yielding massive accuracy gains even with minimal training samples.

A similar effect is observed in 360° video quality assessment, where Chen Li et al. used head-movement and eye-tracking prediction as auxiliary tasks to align a model's internal representations with human perceptual dynamics, ensuring the network evaluates video degradation exactly where a human viewer would actually notice it.

Key Foundational Work

In the Eye of Beholder: Joint Learning of Gaze and Actions in First Person Video (2018) — contributed 2 concept elements to this direction.
Connecting Gaze, Scene, and Attention: Generalized Attention Estimation via Joint Modeling of Gaze and Scene Saliency (2018) — contributed 2 concept elements to this direction.
Towards Making a Dependency Parser See (2019) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_auxiliary_gaze_prediction_d72d2fbd,
  title = {auxiliary gaze prediction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/auxiliary-gaze-prediction/rdr_d72d2fbdd5244c9791759a112b6a2b9a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_auxiliary_gaze_prediction_d72d2fbd_v1,
  title = {auxiliary gaze prediction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/auxiliary-gaze-prediction/rdr_d72d2fbdd5244c9791759a112b6a2b9a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake