If you track the eye movements of a single person watching a video or navigating a room, their gaze path often appears erratic. Humans are prone to idiosyncratic visual wandering—momentarily looking at a random background detail, blinking, or getting distracted by an irrelevant feature.
However, if you record dozens of individuals performing the same task or viewing the same scene, a remarkable statistical phenomenon emerges: individual distractions act as uncorrelated noise, while task-relevant or socially salient features act as a coherent signal. By aggregating these individual gaze patterns, the random noise cancels out, leaving a highly precise map of collective attention.
As Tavakoli et al. demonstrate, eye-tracking data exhibits significantly higher inter-participant consistency than other crowdsourced proxies like mouse tracking, making it an exceptionally stable signal of human intent. Similarly, Palazzi et al. show that while instantaneous driver gaze is highly subjective, aggregating it over a brief temporal window reveals a remarkably consistent, task-driven attentional priority shared across different individuals. Multi-observer gaze aggregation works because it extracts this underlying consensus from a sea of individual behavioral noise.
Geometric and Spatial Aggregation Mechanisms
To mathematically combine separate gazes into a single representation, researchers project individual visual data into a shared coordinate space. The exact mechanism depends on whether the environment is a flat 2D plane or a complex 3D space.
In 2D environments, such as standard videos or images, individual fixations are typically mapped using spatial density estimators. For instance, Gu et al. aggregate gaze points using Gaussian kernels and blob detection to identify clear centroids of shared focus. This transforms scattered points into a unified spatial heatmap, a strategy also leveraged by Fan et al. to localize where multiple agents' sightlines intersect in social scenes.
Individual Gaze Points (Noisy) Gaussian Kernel Smoothing Aggregated Centroid (Signal)
• • ░ ░
• ───────► ░ ▒ ░ ───────► ███
• • ░ ░
When moving to 3D spaces or egocentric perspectives, the geometry shifts from flat coordinates to intersecting volumes. Instead of flat points, gaze is modeled as a 3D vector or a cone-shaped probability field. Park et al. project these individual gaze distributions into a shared 3D coordinate system, creating a volumetric attention field. Where these cones overlap, the spatial density spikes, revealing "gaze concurrences" that objectively mark social saliency or group formations without relying on any single participant's perspective.
This concept extends to tracking fields-of-view (FOV) across multiple mobile users; Essa et al. map multiple egocentric FOVs into a shared reference frame to automatically detect joint attention events during social interactions, such as a group of people looking at the same museum exhibit.
Quantifying Group Dynamics and Synchrony
Aggregation does more than locate targets; it provides a metric for group behavior. By measuring how well a group's gaze aligns—often termed inter-subject correlation (ISC) or attentional synchrony—systems can infer internal states like engagement, comprehension, or expertise.
To quantify this alignment, Sharma et al. introduce a gaze cohesiveness metric that calculates the mean squared error of individual gaze points relative to their collective centroid. When a group is highly engaged, their visual attention synchronizes tightly around shared focal points. This temporal and spatial alignment is a strong indicator of focus; Bühler et al. find that high inter-subject gaze synchrony reliably correlates with self-reported attention during video lectures.
Furthermore, this alignment pattern changes based on domain knowledge and context:
- Expertise: Burling et al. calculate inter-subject correlation to show that experts (such as experienced surveillance operators) exhibit highly coordinated, shared attentional strategies that differ markedly from the scattered gaze of novices.
- Collaboration: In multi-party environments, tracking these shared orientations—even through coarse proxies like head pose, as explored by Wu and Palmer et al.—allows automated systems to map out a group's "common ground" and non-verbal dynamics in real time.
- Clinical Traits: This concept of aggregating across variations to find a stable trait is also valuable in clinical settings. Shic et al. aggregate gaze duration data across diverse social stimuli to reduce individual noise and capture robust, reproducible behavioral biomarkers for autism evaluation.
A Weak Supervisor for Machine Learning
One of the most powerful applications of multi-observer aggregation is generating high-quality training data for deep learning models. Manual annotation of complex visual data (like drawing bounding boxes around salient objects or labeling medical anomalies) is slow, expensive, and prone to human bias. Aggregated gaze provides a natural, passive alternative.
Because the aggregation process filters out individual noise, the resulting consensus map acts as a robust semantic prior. Karthikeyan et al. utilize multi-viewer gaze data as a weak supervisor to automatically segment attentionally important objects in videos, bypassing traditional motion cues.
In computer vision for autonomous driving, Xia et al. demonstrate that training networks on multi-focus attention maps—which highlight all potential hazards simultaneously—prevents models from falling into common loss-function traps (like over-indexing on the center of the road) and forces them to prioritize critical, rare events like crossing pedestrians.
[Raw Video Frame] ───► [Multiple Human Drivers' Gaze] ───► [Aggregated Attention Map]
│
▼
[Weakly Supervised Loss]
▲
[Model Prediction] ────────────────────────────────────────────────┘
This approach scales remarkably well to highly specialized domains where human annotation is particularly difficult. In digital pathology, Gu et al. show that aggregating eye-tracking data from multiple pathologists filters out individual fixations on distractors. This collective signal produces training labels for mitosis detection that allow a convolutional neural network to match the performance of models trained on tedious, manually drawn ground truths.
The Predictability of Attentional Consensus
If collective attention is so stable, can we predict when a group will achieve consensus and where they will look without collecting eye-tracking data every time?
The evidence suggests that inter-subject consistency is not a random variable, but an inherent property of the visual stimulus itself. Volokitin et al. demonstrate that high-level semantic features extracted from deep neural networks can accurately predict how much a group of human observers will agree on an image's salient regions. In immersive environments, Li et al. show that human head and eye movements in 360° videos are highly consistent across users, allowing automated models to reliably predict human perception based on the video content alone.
Beyond static semantics, high-level structural and stylistic choices also dictate consensus. Bruckert et al. incorporate "cinematic priors"—such as shot size and camera motion—to model why gaze tightly clusters during certain movie scenes and scatters during others.
Finally, this aggregation can even be performed across demographic cohorts rather than simultaneous viewers. Zhang et al. introduce cohort-based attention aggregation, merging individual saliency maps from patients sharing specific clinical attributes to isolate consistent, population-wide structural markers of aging and degeneration on medical radiographs. Whether analyzing a cinematic cut, a crowded room, or a medical image, aggregating multiple viewpoints uncovers a shared visual truth that no single observer could reveal alone.
Key Foundational Work
3D Social Saliency from Head-mounted Cameras (2012) — contributed 3 concept elements to this direction.
Prediction of the inter-observer visual congruency (IOVC) and application to image ranking (2011) — contributed 3 concept elements to this direction.