Attention Flow: End-to-End Joint Attention Estimation

Ömer Sümer, Peter Gerjets, Ulrich Trautwein, Enkelejda Kasneci

TL;DR

Attention Flow is a novel end-to-end framework designed to estimate "joint attention"—the shared gaze of multiple people on a single object—directly from raw video frames. By combining saliency-augmented training maps with specialized channel-wise and spatial attention mechanisms, the model outperforms complex multi-stage pipelines without requiring external face or head detectors at test time.

What Problem Does This Paper Address?

In social settings, humans constantly engage in joint attention, whether it's students looking at a teacher’s demonstration or a group of friends laughing at the same phone screen. For computer vision, detecting these moments is incredibly difficult.

Traditional approaches are "heavy": they usually require a multi-step pipeline that detects every head, estimates individual gaze vectors, and then calculates where those vectors intersect. These pipelines are prone to "error propagation"—if the head detector fails, the whole system fails. Furthermore, "saliency" (what a third-person observer finds interesting) is often confused with "joint attention" (what the people inside the scene are actually looking at). The authors aim to bridge this gap with a streamlined, end-to-end model that understands social context directly from pixels.

Methods

The authors introduce Attention Flow, a deep learning architecture composed of a feature encoder (ResNet-50/DRN-A-50) and a generator that produces a two-channel heatmap. One channel predicts face locations, while the other predicts the likelihood of a joint attention point.

The core innovations include:

  • Saliency-Augmented Pseudo-Attention Maps: During training, the authors don't just use binary labels. They create "pseudo-attention maps" by combining Gaussian distributions of head/object locations with saliency maps generated by Deep Gaze II. This helps the model learn the relative importance of different regions in a social scene.
  • Channel-wise Attention: This module performs feature selection on the encoder’s output. It weights different feature channels based on their contribution to the task, essentially "filtering" the deep features to focus on those most relevant to social gaze.
  • Spatial Attention: This acts as a refinement step on the final heatmap, helping the model pinpoint the exact center of joint attention more accurately.
  • End-to-End Inference: At test time, the model requires only the raw image. It does not need pre-computed head bounding boxes or gaze vectors, making it significantly faster and more robust than modular pipelines.

Datasets

The model was trained and evaluated on the VideoCoAtt dataset.

  • Composition: 380 RGB video sequences from 20 different TV shows.
  • Scale: Over 250,000 training frames and 113,000 testing frames.
  • Complexity: It features diverse social formations, varying lighting, and complex backgrounds, making it a rigorous benchmark for "in-the-wild" joint attention.

Key Insights and Ideas

  • Saliency as a Scaffold: The authors found that while saliency maps don't perfectly represent joint attention, they are an excellent "precursor." Using them to augment training labels makes the high-level inference task much more learnable for a neural network.
  • Feature Selection is Crucial: The channel-wise attention module was a major winner. It reduced the mean $L_2$ error significantly (from ~70 to ~62), suggesting that specific deep features are much more "socially informative" than others.
  • Single-Frame Strength: Surprisingly, this single-frame model outperformed several state-of-the-art models that use temporal information (like LSTMs). This suggests that the spatial arrangement of faces and objects in a single "social snapshot" contains a massive amount of information that previous models weren't fully exploiting.

Key Results

The Attention Flow model set new benchmarks on the VideoCoAtt dataset:

  • Prediction Accuracy: Achieved 78.1%, outperforming the previous best "Raw Image" approach (52.3%) and even complex models using gaze and region proposals (68.5%).
  • Localization Error: Achieved an $L_2$ distance of 62.84, which is a significant improvement over standard gaze-following baselines.
  • Qualitative Success: The model successfully localized joint attention even in groups of 3–4 people, handling both frontal and side-profile faces effectively.

Limitations and Future Work

The authors acknowledge a few hurdles:

  • Wide Angles and Distance: The model struggles when scenes are very broad or when faces are far from the camera, making it difficult to discern subtle gaze cues.
  • Dataset Bias: The VideoCoAtt dataset consists of "acted" TV scenes. The authors note that real-world social formations (like those in a classroom or therapy session) might differ in structure.
  • Future Directions: The team suggests specializing the model for specific domains, such as automated educational assessment or developing training curricula for children with autism, where automated joint attention analysis could be transformative.

Summary updated

Abstract

This paper addresses the problem of understanding joint attention in third-person social scene videos. Joint attention is the shared gaze behaviour of two or more individuals on an object or an area of interest and has a wide range of applications such as human-computer interaction, educational assessment, treatment of patients with attention disorders, and many more. Our method, Attention Flow, learns joint attention in an end-to-end fashion by using saliency-augmented attention maps and two novel convolutional attention mechanisms that determine to select relevant features and improve joint attention localization. We compare the effect of saliency maps and attention mechanisms and report quantitative and qualitative results on the detection and localization of joint attention in the VideoCoAtt dataset, which contains complex social scenes.
BibTeX
@inproceedings{sumer2020attention,
  author = {Ömer Sümer and Peter Gerjets and Ulrich Trautwein and Enkelejda Kasneci},
  title = {{Attention Flow: End-to-End Joint Attention Estimation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=silGmtUnXGJ},
}

Loading related content...

Figure