AAN: Attributes-Aware Network for Temporal Action Detection

Rui Dai, Srijan Das, Michael S. Ryoo, François BrémondarXiv 2023 · DOI · arXiv

Introduction

Temporal Action Detection (TAD) is a fundamental challenge in computer vision that requires models to identify both the category and the precise temporal boundaries of actions within long, untrimmed videos. While deep learning has achieved high performance on short, trimmed video clips, long-term video understanding remains difficult due to the complexity of human-object interactions and the varying durations of activities.

Existing methodologies generally follow two paths: 3D Convolutional Neural Networks (CNNs), which excel at motion capture but lack explicit object-centric modeling, and object-detector pipelines, which are computationally expensive and limited by predefined categories. The Attributes-Aware Network (AAN) addresses these limitations by leveraging the rich semantic space of OpenAI’s CLIP to "mine" object-centric information without the overhead of traditional object detectors.

Methodology

The AAN framework introduces a novel approach to feature representation by disentangling specific object attributes from global image features. The architecture is divided into three primary stages:

1. Feature Extraction

The system utilizes a frozen CLIP visual encoder to generate frame-level features. By keeping the encoder frozen, the model benefits from the broad semantic knowledge captured during CLIP’s large-scale pre-training while maintaining computational efficiency.

2. Attributes Extractor

Instead of employing a bounding-box detector, AAN uses "semantic anchors." The authors define a set of $N$ object names relevant to the target dataset (e.g., "chair," "bottle," "knife"). These names are processed through the CLIP Text encoder to produce pure semantic vectors. The Attributes Extractor then learns to project the global visual features into $N$ specific attribute features. This process is guided by an attribute loss (MSE) that ensures the extracted features remain aligned with the CLIP semantic anchors, effectively isolating relevant object information from the background.

3. Graph Reasoning Block

Once attributes are extracted for each frame, they are treated as nodes in a graph to model complex interactions:

  • Inter-attribute Reasoning: A multi-head self-attention module analyzes how different objects relate within a single frame (e.g., the spatial-semantic relationship between a person and a tool).
  • Temporal Reasoning: Temporal convolutions are applied to model how these relationships evolve over the duration of the video.

Experimental Results

The AAN model was evaluated on two benchmarks known for their complexity in daily living activities: Charades and Toyota Smarthome Untrimmed (TSU).

Performance Metrics

AAN established new state-of-the-art (SOTA) results on both datasets:

  • Charades: The model achieved 32.0% mAP, significantly outperforming the previous best of 28.8% (ViVit-L + TTM). This is particularly notable given the multi-label nature of Charades, where multiple actions often overlap.
  • TSU: The model reached 41.3% mAP, a substantial improvement over the previous benchmark of 33.7% set by MS-TCT.

Qualitative Analysis

The model demonstrates superior performance in "object-dominated" actions. For instance, AAN showed a +37.2% improvement in detecting "Closing a window" and a +26.4% improvement for "Sitting on a chair" compared to baseline models. This confirms that explicit attribute modeling is highly effective for actions defined by specific human-object interactions.

Key Insights

  1. Disentanglement vs. Detection: The success of AAN suggests that precise spatial localization (bounding boxes) is not strictly necessary for action detection. Identifying the presence and relevance of an object ("what-ness") through semantic disentanglement is often sufficient and more efficient.
  2. The Importance of Reasoning: Ablation studies indicate that attribute extraction alone provides only a marginal boost (+1.4% mAP). The significant performance gains (+14% mAP) are realized only when the Graph Reasoning block is introduced, highlighting that the interaction between objects over time is the critical factor in TAD.
  3. CLIP as a Foundation: AAN outperforms standard temporal models even when using identical backbones, proving that structured attribute modeling is superior to generic feature learning for complex video tasks.

Conclusion and Future Work

The Attributes-Aware Network provides a scalable and efficient framework for temporal action detection by bridging the gap between global visual features and fine-grained object semantics. While the current model relies on a predefined vocabulary of attributes, future iterations may explore fully open-vocabulary inference. The authors suggest that this method of disentangling CLIP features via semantic anchors holds potential for various other video understanding tasks beyond action detection.

Summary updated

Abstract

The challenge of long-term video understanding remains constrained by the efficient extraction of object semantics and the modelling of their relationships for downstream tasks. Although the CLIP visual features exhibit discriminative properties for various vision tasks, particularly in object encoding, they are suboptimal for long-term video understanding. To address this issue, we present the Attributes-Aware Network (AAN), which consists of two key components: the Attributes Extractor and a Graph Reasoning block. These components facilitate the extraction of object-centric attributes and the modelling of their relationships within the video. By leveraging CLIP features, AAN outperforms state-of-the-art approaches on two popular action detection datasets: Charades and Toyota Smarthome Untrimmed datasets.
BibTeX
@misc{dai2023aan,
  author = {Rui Dai and Srijan Das and Michael S. Ryoo and François Brémond},
  title = {{AAN: Attributes-Aware Network for Temporal Action Detection}},
  year = {2023},
  eprint = {2309.00696},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.00696},
}
Figure