Abstract
We address the problem of detecting actions, such as drinking or opening a door, in hours of challenging video data. We propose a model based on a sequence of atomic action units, termed “actoms”, that are characteristic for the action. Our model represents the temporal structure of actions as a sequence of histograms of actom-anchored visual features. Our representation, which can be seen as a temporally structured extension of the bag-of-features, is flexible, sparse and discriminative. We refer to our model as Actom Sequence Model (ASM). Training requires the annotation of actoms for action clips. At test time, actoms are detected automatically, based on a non parametric model of the distribution of actoms, which also acts as a prior on an action's temporal structure. We present experimental results on two recent benchmarks for temporal action detection, “Coffee and Cigarettes” and the DLSBP dataset. We show that our ASM method outperforms the current state of the art in temporal action detection.
Introduction
Action recognition in computer vision is typically divided into two distinct challenges: classification (identifying an action in a pre-trimmed clip) and detection (locating the start and end of an action within hours of unsegmented video). Traditional approaches have relied heavily on the "bag-of-features" (BoF) model. While effective for general classification, BoF is inherently orderless, ignoring the timing of visual patterns. This lack of temporal structure makes it difficult to distinguish between actions with similar motions but different sequences, such as "sitting down" versus "standing up."
Presented at CVPR 2011, the Actom Sequence Model (ASM) addresses these limitations by decomposing actions into their constituent "atoms" and modeling their temporal progression.
The Actom Sequence Model (ASM)
The core innovation of this framework is the actom—a semantically meaningful key moment within an action. For instance, a "drinking" action can be broken down into picking up a glass, the glass touching the lips, and returning the glass to the table.
Temporal Structuring and Adaptive Radii
Unlike global BoF histograms, the ASM creates a sequence of histograms, each anchored at a specific actom timestamp. A temporally weighted assignment scheme is employed, where features closer to the actom's center are weighted more heavily using a Gaussian window. To account for variations in action speed, the model utilizes an adaptive radius for each actom, scaling its temporal reach based on the distance to neighboring units.
For detection in novel videos, the system employs a non-parametric generative model (via Kernel Density Estimation) to predict likely inter-actom spacings, allowing the model to naturally handle actions performed at different speeds without exhaustive sliding-window searches.
Experimental Evaluation
The authors evaluated the ASM on two challenging benchmarks: the "Coffee and Cigarettes" dataset, focusing on drinking and smoking, and the "DLSBP" dataset, which includes actions like "OpenDoor" and "SitDown" across 15 different films.
Key Results
The ASM consistently outperformed standard BoF and rigid temporal grids. On the "Coffee and Cigarettes" drinking task, ASM achieved 57% Average Precision (AP), a significant jump from the 40% state-of-the-art at the time.
To further validate the model's precision, the authors introduced the OVAA (Overlap All Actoms) metric, which requires a detection to encompass all key actoms of an action. Under this stricter evaluation, the ASM maintained a 50% AP for drinking, while the BoF baseline dropped to just 11%. This demonstrates that the ASM is truly identifying the action's structure rather than just picking up on surrounding environmental context.
Conclusion
By prioritizing temporal structure over simple statistics, the Actom Sequence Model provides a robust framework for short action detection. While it currently requires manual actom annotation during training, it offers a more precise and flexible alternative to traditional sliding-window BoF methods. Future work may involve extending this atomic approach to longer, more complex hierarchical activities.
Summary updated