Coarse Temporal Attention Network (CTA-Net) for Driver's Activity Recognition

Zachary Wharton, Ardhendu Behera, Yonghuai Liu, Nik BessisWACV 2021

TL;DR

This paper introduces CTA-Net, a novel deep learning framework for recognizing subtle driver activities, which are challenging due to similar body movements. CTA-Net employs a "glimpse sensor" with self-attention to capture fine-grained spatial changes and a coarse temporal attention mechanism to model high-level temporal relationships like "before," "during," and "after" actions, achieving state-of-the-art results on multiple datasets.

What Problem Does This Paper Address?

Recognizing human activities from videos has seen significant progress, but traditional methods often struggle with activities that involve subtle body movements and similar patterns, such as those performed by drivers. Unlike highly distinctive actions (e.g., running, jumping), driver activities like eating, drinking, or texting often involve the same subject with minor variations in body parts and object interactions. This makes it difficult to distinguish between them. Furthermore, understanding driver behavior is crucial for developing advanced driver-assistance systems (ADAS), autonomous driving, and ensuring driving safety. Existing approaches often adapt general action recognition models, but they don't fully capture the nuanced temporal dynamics and subtle spatial changes inherent in driver activities.

Methods

The proposed CTA-Net framework is built around two key novel components:

  • Glimpse Sensor with Coarse Temporal Attention: This module uses a modified ResNet-50 as a "glimpse sensor" to extract features from video frames. Crucially, it incorporates three branches that focus on coarse temporal bandwidths (inspired by "before," "during," and "after" temporal relationships). This allows the model to capture high-level temporal context, assuming that an action unfolds across these phases (e.g., reaching for a phone, using it, and putting it away). Within each branch, a self-attention mechanism is employed to highlight important spatial changes within specific temporal segments.
  • Temporal Attention Mechanism: After the glimpse sensor extracts features, these are fed into an LSTM to model temporal dependencies. The novelty here lies in the temporal attention mechanism, which learns to weigh the importance of different hidden states of the LSTM. Instead of relying on raw image features or focusing solely on specific body parts (like hands), this attention mechanism selectively focuses on the LSTM's internal states to generate a more contextually rich representation for activity recognition. This allows the model to adaptively decide how much to attend to past observations when inferring the current activity.

Datasets

The authors evaluated their CTA-Net on four publicly accessible datasets:

  • Drive&Act [31]: A large-scale dataset with over 9.6 million frames, featuring 12 coarse-level activities and 34 fine-grained semantic actions.
  • Distracted Driver V1 [1]: Contains 10 activity classes, with both training and testing sets provided.
  • Distracted Driver V2 [8]: A newer iteration of Distracted Driver V1, also with 10 activity classes.
  • SBU Kinect Interaction [57]: A dataset of traditional human activities, used to demonstrate the broader applicability of the model beyond driver-specific tasks.

Key Insights and Ideas

  • Temporal Relationships Matter for Subtle Actions: The paper strongly suggests that explicitly modeling "before," "during," and "after" temporal phases is critical for distinguishing between similar driver activities. This goes beyond simply looking at frame-by-frame content.
  • Self-Attention for Spatial Nuances: The integration of self-attention within the glimpse sensor effectively captures subtle spatial changes that are key discriminators for driver actions.
  • LSTM Hidden States as Rich Context: The temporal attention mechanism's ability to selectively focus on LSTM hidden states proves more effective than relying solely on visual features or hard attention on specific body parts for driver activity recognition.
  • RGB-Only Performance: The model achieves state-of-the-art results using only RGB video, highlighting its efficiency and reducing the need for multi-modal sensors (like depth or pose).

Key Results

CTA-Net achieved state-of-the-art performance across the evaluated datasets:

  • Drive&Act: Outperformed existing methods by a significant margin, achieving 72.42% accuracy on fine-grained tasks (validation) and 65.25% (testing), and 62.82% (validation) and 52.31% (testing) on coarse tasks. This represents an 18.2% improvement for coarse tasks compared to the previous best.
  • Distracted Driver V1: Achieved 84.09% accuracy, significantly outperforming prior work.
  • Distracted Driver V2: Achieved 92.50% accuracy.
  • SBU Kinect Interaction: Achieved 92.9% accuracy, outperforming RGB-only methods and being competitive with multi-modal approaches.

The ablation studies confirmed that both the coarse temporal relationships and the novel attention mechanism significantly contribute to the model's performance.

Limitations and Future Work

  • Confusion with Minimal Actions: The paper acknowledges that activities involving very little visual action (e.g., pressing a button, "watching videos") or small objects (e.g., "taking off sunglasses") can still be confused with other actions.
  • Future Work: The authors suggest applying the proposed technique for the development of a driving assistance system, implying a move towards real-time application and integration into vehicle systems.

Summary updated

BibTeX
@inproceedings{wharton2021coarse,
  author = {Zachary Wharton and Ardhendu Behera and Yonghuai Liu and Nik Bessis},
  title = {{Coarse Temporal Attention Network (CTA-Net) for Driver's Activity Recognition}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
}
Figure