C3D and Localization Model for Locating and Recognizing the Actions from Untrimmed Videos (Student Abstract)

Himanshu Singh, Tirupati Pallewad, Badri N. Subudhi, Vinit JakhetiyaAAAI 2022 · DOI

Introduction

In the field of computer vision, recognizing actions in "trimmed" videos—where the clip is perfectly aligned with the start and end of an activity—is a relatively mature task. However, real-world video data is typically "untrimmed," containing significant background noise, irrelevant movements, and multiple interleaved actions. This presents a dual challenge: Temporal Action Localization (identifying the precise start and end timestamps) and Action Recognition (classifying the specific action).

Researchers from IIT Jammu have proposed a framework to address these challenges by efficiently extracting spatio-temporal features and defining action boundaries without the need for exhaustive frame-by-frame manual annotations.

Methodology

The proposed architecture utilizes a multi-stage pipeline designed to capture motion dynamics and segment videos into meaningful units.

The proposed multi-stage pipeline for action localization and recognition.

The process begins with feature extraction using a pre-trained 3D Convolutional Neural Network (C3D), which captures motion by processing stacks of frames simultaneously. The core innovation lies in Action-Bytes Segmentation. The authors posit that while high-level deep features evolve slowly, abrupt shifts in the feature space signal action transitions. By mapping high-dimensional C3D features into a latent space, the model identifies these "spikes" to decompose the video into "action-bytes."

To train without exhaustive labels, the system uses an iterative clustering process (Partitioning Around Medoids) to generate "pseudo-labels." The final model employs a KNN classifier and a localization module optimized via a joint loss function: Multiple Instance Learning (MIL) loss for classification and co-activity similarity loss for localization. This synergy allows the model to learn that knowing what an action looks like helps determine where it occurs.

Experimental Results

The framework was evaluated on the Thumos14 benchmark, which consists of thousands of untrimmed videos across 20 sport-related classes.

Action Classification

The proposed method achieved a remarkable 96.10% mean Average Precision (mAP) in action classification. As shown in the comparison below, this significantly outperforms established state-of-the-art (SOTA) techniques such as "Two Stream" and "Objects + Motion."

Comparison of action classification mAP on the Thumos14 dataset.

Action Localization

In terms of temporal localization, the model was evaluated using Intersection over Union (IoU) thresholds. At a strict IoU of 0.5, the model achieved a score of 22.54. While this trails slightly behind the SSN method (29.1), it outperforms ten other major SOTA methods, demonstrating competitive performance in pinpointing action boundaries.

Conclusion and Future Work

The "Action-Byte" philosophy offers a robust way to handle the inherent fuzziness of action boundaries in untrimmed footage. By leveraging latent space shifts and synergistic loss functions, the model achieves high accuracy in both identification and localization. Future research will focus on refining hyperparameters to further boost localization precision and exploring the system's ability to handle "unknown" action classes.

Summary updated

BibTeX
@inproceedings{singh2022c,
  author = {Himanshu Singh and Tirupati Pallewad and Badri N. Subudhi and Vinit Jakhetiya},
  title = {{C3D and Localization Model for Locating and Recognizing the Actions from Untrimmed Videos (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i11.21662},
}
Figure