Abstract
This paper addresses recognition of human activities with stochastic structure, characterized by variable space-time arrangements of primitive actions, and conducted by a variable number of actors. Our approach classifies the activity of interest as well as identifies the relevant foreground in the video. Each activity representation is considered as a mixture distribution of BoWs captured by a Sum-Product Network (SPN). In our approach, SPN represents a linear mixture of many bags-of-words (BoWs) where each BoW represents an important foreground part of the activity. This mixture distribution is efficiently computed by organizing the BoWs in a hierarchy, where children BoWs are nested within parent BoWs. SPN allows us to model this mixture since it consists of terminal nodes representing BoWs, product nodes, and sum nodes organized in a number of layers. The products are aimed at encoding particular configurations of primitive actions, and the sums serve to capture their alternative configurations. SPN inference amounts to parsing the SPN graph, which yields the most probable explanation (MPE) of the video foreground. SPN inference has linear complexity in the number of nodes, under fairly general conditions, enabling fast and scalable recognition. The connectivity of SPN and the parameters of BoW distributions are learned under weak supervision using a variational EM algorithm. For our evaluation, we have compiled and annotated a new Volleyball dataset. Our classification accuracy and localization results are superior to those of the state of the art on current benchmarks as well as our Volleyball datasets.
Introduction
Parsing video to recognize co-occurring individual and group activities is a computationally intensive task. Modern computer vision models often utilize Spatiotemporal AND-OR Graphs (ST-AOG), which are highly expressive hierarchical grammars. These models can represent how a low-level object relates to an individual action, which in turn constitutes a broader group activity over time.
However, the expressiveness of ST-AOGs comes with a significant computational burden. Traditional inference requires running a massive battery of object detectors, action classifiers, and trackers across every frame of a video. This "cost-insensitive" approach is often prohibitively expensive for real-world applications. This paper proposes a novel framework that treats video parsing as a resource-scheduling problem, using Monte Carlo Tree Search (MCTS) to optimize the sequence of computational processes.
Methodology: Cost-Sensitive Inference
The authors propose a cost-sensitive inference framework that views video parsing as an open-loop planning problem. The goal is to find the most informative sequence of computer vision algorithms to run within a limited time budget.
The ST-AOG Model
The framework utilizes a hierarchical grammar representing activities at three distinct levels:
- Group Activities (Level 1)
- Individual Actions (Level 2)
- Objects (Level 3)
To navigate this graph, four types of computational "moves" or processes are defined:
- $\alpha$ process: Direct detection from raw video features.
- $\beta$ process: Bottom-up prediction (e.g., using detected objects to predict an action).
- $\gamma$ process: Top-down context (e.g., using group activity context to refine object detection).
- $\omega$ process: Temporal tracking across time intervals.
MCTS Scheduling
Instead of executing every possible process, the system uses MCTS to identify the optimal sequence of these moves. MCTS explores the tree of possible inference actions using the UCT (Upper Confidence Bound applied to Trees) algorithm. This approach balances the exploration of new sequences with the exploitation of known high-performing ones. The search aims to maximize the expected log-likelihood of the resulting parse graph, ensuring that the most "informative" detectors are prioritized.
Experimental Results
The model was evaluated on several challenging benchmarks, including the UCLA Courtyard Dataset, which features complex, overlapping activities in high-resolution video, and the Collective Activity Dataset.
Key Findings
- Efficiency Gains: The MCTS-scheduled inference achieved up to a 100x speedup compared to standard cost-insensitive methods with minimal loss in accuracy.
- Accuracy: On the UCLA Courtyard dataset, the MCTS approach significantly outperformed previous Reinforcement Learning (Q-learning) based scheduling methods in both precision and false-positive rates.
- Graceful Degradation: The system demonstrates "graceful degradation," meaning that as the time budget decreases, it still provides the most probable parse possible within the allowed constraints rather than failing completely.
- Temporal Coherence: By accounting for temporal relations (the $\omega$ process) at all semantic levels, the model achieved superior Match Error Correction Rates (MECR) compared to existing unified frameworks.
Conclusion
By reframing computer vision inference as a resource-allocation and planning problem, this research demonstrates that complex activity recognition can be made significantly more efficient. The use of MCTS allows the system to bypass the limitations of Markovian assumptions, accounting for higher-order dependencies across the entire inference sequence. Future work may explore hybrid closed-loop systems to further enhance performance in highly unpredictable environments.
Summary updated