Spatiotemporal Deformable Models for Long-Term Complex Activity Detection

Salman Khan, Fabio Cuzzolin

Introduction

In the field of computer vision, action recognition has traditionally focused on "atomic actions"—short, discrete movements such as "stepping" or "cutting." However, real-world applications, particularly in autonomous driving and robotic surgery, require the detection of "complex activities." These are high-level, multi-step events composed of numerous atomic actions over extended durations. For instance, "negotiating an intersection" involves a sequence of moving, stopping for signals, yielding to pedestrians, and resuming motion.

Current methodologies often struggle with these long-term activities because they typically analyze short video snippets or fail to model the evolving spatial and temporal relationships between various actors. This research introduces a framework designed to bridge this gap by capturing the flexible geometry and semantic structure of complex activities in dynamic environments.

Methodology: A Three-Stage Pipeline

The proposed framework utilizes a three-stage pipeline to transition from raw pixels to high-level complex semantics.

Overview of the proposed framework, illustrating the transition from input video to action tube detection, parts-based deformation, graph representation, and final activity localization.

1. Action Tube Detection

The process begins with identifying "action tubes"—sequences of bounding boxes tracked over time—for atomic actions. Using a pre-trained detector (AMTNet), the system generates a fixed-size representation of the constituent "parts" of a complex activity.

2. 3D Deformable RoI Pooling

A key architectural innovation of this work is the 3D Deformable RoI Pooling layer. Inspired by 2D deformable models, this layer learns the flexible spatiotemporal geometry of action tubes. By generating offsets for the pooling grid, the model can adapt to different execution "styles" of an action (e.g., a wide vs. sharp turn) or variations in camera viewpoint. This ensures that the extracted features are invariant to the specific manner in which an agent performs an action.

Illustration of parts-based deformation within detected action tubes, showing the learned flexible geometry used for feature extraction.

3. Spatiotemporal Scene Graphs and GCNs

The detected action tubes are treated as nodes in a spatiotemporal scene graph. Edges are constructed based on three semantic relationships:

  • Temporal Order: Capturing the sequence of appearance.
  • Action Labels: Grouping similar types of atomic actions.
  • Feature Similarity: Connecting tubes with similar visual or behavioral characteristics.

A Graph Convolutional Network (GCN) processes this graph to extract global context. Finally, a softmax classifier combined with a sliding-window mechanism is used to recognize and localize the complex activity within the video stream.

Experimental Evaluation

The framework was evaluated on two challenging real-world datasets, augmented with new "complex activity" annotations:

  • ROAD (Road Event Awareness Dataset): Focused on autonomous driving, with labels for activities like "Negotiating pedestrian crossing" and "Merging into vehicle lane."
  • SARAS-ESAD (Surgical Action Detection Dataset): Focused on robotic surgery, involving the annotation of contiguous surgical phases such as "Bladder neck transection."

Key Results

The proposed method consistently outperformed state-of-the-art graph-based competitors, such as P-GCN and G-TAD. On the ROAD dataset, the model achieved a mean Average Precision (mAP) of 59.4% at an IoU threshold of 0.5, significantly higher than G-TAD (48.7%) and P-GCN (46.4%). Similarly, on the SARAS-ESAD dataset, it reached 51.5% mAP, surpassing the competition.

The model demonstrated particular strength in high-frequency activities, such as "Waiting in a queue" (89.8% mAP). However, performance on rare events like "Sudden appearance" remains a challenge, highlighting the difficulties posed by long-tail data distributions.

Conclusion and Future Work

This research demonstrates that complex activities are best understood as ensembles of atomic actions with high-level semantic meaning. By incorporating 3D deformable pooling and relational scene graphs, the model effectively captures both the "how" (geometry) and the "what" (semantics) of dynamic scenes.

Future work will focus on developing more robust action tube detectors, as the current pipeline's performance is heavily dependent on the initial detection stage. Additionally, the researchers aim to explore end-to-end training and the inclusion of non-action elements—such as static objects and environmental attributes (e.g., traffic light states)—to create a more holistic representation of complex environments.

Summary updated

BibTeX
@inproceedings{khan2021spatiotemporal,
  author = {Salman Khan and Fabio Cuzzolin},
  title = {{Spatiotemporal Deformable Models for Long-Term Complex Activity Detection}},
  booktitle = {BMVC},
  year = {2021},
  url = {https://openreview.net/forum?id=NZHUlqLxjf},
}
Figure