Attention Flow: End-to-End Joint Attention Estimation

Ömer Sümer, Peter Gerjets, Ulrich Trautwein, Enkelejda Kasneci

Abstract

This paper addresses the problem of understanding joint attention in third-person social scene videos. Joint attention is the shared gaze behaviour of two or more individuals on an object or an area of interest and has a wide range of applications such as human-computer interaction, educational assessment, treatment of patients with attention disorders, and many more. Our method, Attention Flow, learns joint attention in an end-to-end fashion by using saliency-augmented attention maps and two novel convolutional attention mechanisms that determine to select relevant features and improve joint attention localization. We compare the effect of saliency maps and attention mechanisms and report quantitative and qualitative results on the detection and localization of joint attention in the VideoCoAtt dataset, which contains complex social scenes.

Introduction

In social settings, humans constantly engage in joint attention—whether students are observing a teacher’s demonstration or a group of friends is focused on a single screen. For computer vision, detecting these moments is notoriously difficult. Traditional approaches typically rely on "heavy" multi-step pipelines that detect heads, estimate individual gaze vectors, and calculate intersections. These systems are prone to error propagation; if a head detector fails, the entire estimation collapses. Furthermore, general visual saliency (what an observer finds interesting) is often conflated with joint attention (what participants in the scene are looking at).

To address these issues, we propose Attention Flow, a streamlined, end-to-end model that understands social context directly from pixels, bypassing the need for external detectors at test time.

Methods

The Attention Flow architecture is composed of a feature encoder (ResNet-50 or DRN-A-50) and a generator that produces a two-channel heatmap: one for face locations and another for the likelihood of a joint attention point.

The Attention Flow architecture, featuring a feature encoder, generator, and a saliency-augmented co-attention map generation module.

The framework introduces several core innovations:

  • Saliency-Augmented Pseudo-Attention Maps: During training, the model utilizes "pseudo-attention maps" that combine Gaussian distributions of head/object locations with saliency maps generated by Deep Gaze II. This helps the model learn the relative importance of different regions within a social scene.
  • Channel-wise Attention: This module performs feature selection on the encoder’s output, weighting channels based on their contribution to social gaze estimation.
  • Spatial Attention: This mechanism acts as a refinement step on the final heatmap, helping the model pinpoint the exact center of joint attention more accurately.
  • End-to-End Inference: At test time, the model requires only the raw image, making it significantly faster and more robust than modular pipelines.

Experimental Evaluation

The model was trained and evaluated on the VideoCoAtt dataset, which consists of 380 RGB video sequences from 20 different TV shows. With over 360,000 total frames, the dataset features diverse social formations, varying lighting, and complex backgrounds, providing a rigorous benchmark for "in-the-wild" joint attention estimation.

Results and Discussion

Attention Flow set new benchmarks on the VideoCoAtt dataset, achieving a prediction accuracy of 78.1%. This significantly outperforms the previous best "Raw Image" approach (52.3%) and even complex models using explicit gaze and region proposals (68.5%). The model also achieved a mean $L_2$ localization error of 62.84.

Qualitative results showing (a) input image, (b) saliency estimation, (c) face likelihood, and (d) the final co-attention likelihood map.

Key insights from our experiments include:

  • Saliency as a Scaffold: While saliency maps do not perfectly represent joint attention, they serve as an excellent precursor, making the high-level inference task more learnable.
  • Feature Selection: The channel-wise attention module was crucial, reducing the mean $L_2$ error significantly by focusing the network on socially informative deep features.
  • Spatial arrangement: The model's success suggests that the spatial arrangement of faces and objects in a single "social snapshot" contains sufficient information for high-accuracy estimation, often outperforming models that rely on temporal information.

Limitations and Future Work

Despite its success, the model faces challenges in wide-angle scenes where faces are far from the camera, making subtle gaze cues difficult to discern. Future research will focus on specializing the model for specific domains, such as automated educational assessment or therapeutic tools for children with attention disorders, where automated joint attention analysis could be transformative.

Summary updated

Abstract

This paper addresses the problem of understanding joint attention in third-person social scene videos. Joint attention is the shared gaze behaviour of two or more individuals on an object or an area of interest and has a wide range of applications such as human-computer interaction, educational assessment, treatment of patients with attention disorders, and many more. Our method, Attention Flow, learns joint attention in an end-to-end fashion by using saliency-augmented attention maps and two novel convolutional attention mechanisms that determine to select relevant features and improve joint attention localization. We compare the effect of saliency maps and attention mechanisms and report quantitative and qualitative results on the detection and localization of joint attention in the VideoCoAtt dataset, which contains complex social scenes.
BibTeX
@inproceedings{sumer2020attention,
  author = {Ömer Sümer and Peter Gerjets and Ulrich Trautwein and Enkelejda Kasneci},
  title = {{Attention Flow: End-to-End Joint Attention Estimation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=silGmtUnXGJ},
}
Figure