TL;DR
This paper introduces Stacked Spatio-Temporal Graph Convolutional Networks (Stacked-STGCN) for action segmentation in long videos. By extending existing STGCNs to handle diverse node types (scene, actor, object, action) and flexible temporal connections, they achieve state-of-the-art performance on complex action segmentation tasks.
What Problem Does This Paper Address?
Action segmentation, the task of identifying and localizing specific actions within a video sequence, is a challenging problem. Existing methods often struggle with long videos, complex activities involving multiple actors and objects, and the need to precisely pinpoint action boundaries. While graph neural networks (GNNs) have shown promise, previous approaches like STGCN were primarily designed for skeleton-based action recognition and had limitations in handling the rich, heterogeneous contextual information present in real-world scenarios, as well as the dynamic nature of complex activities where objects and actors can appear and disappear. This paper aims to bridge this gap by proposing a more flexible and comprehensive graph-based approach for action segmentation.
Methods
The core innovation is the Stacked Spatio-Temporal Graph Convolutional Network (Stacked-STGCN). This architecture builds upon the Spatio-Temporal Graph Convolutional Network (STGCN) but introduces several key advancements:
- Heterogeneous Nodes and Flexible Connections: Unlike previous STGCNs that focused solely on human skeleton joints, Stacked-STGCN allows for nodes representing various entities like scenes, actors, objects, and actions. Crucially, it supports flexible temporal connections that can span multiple time steps, enabling it to model complex, non-sequential relationships. This also allows for arbitrary edge connections to account for graph deformations caused by missed detections or occlusions.
- Stacked Hourglass Architecture: The authors adapt the successful hourglass network architecture, typically used for grid-like data, to graph data. This encoder-decoder structure with repeated downsampling and upsampling, combined with skip connections, allows for learning features at multiple scales, leading to improved generalization and localization accuracy.
- Varied Feature Length Handling: Recognizing that different node types require descriptors of varying lengths, the Stacked-STGCN can either use additional convolutional layers to standardize feature lengths or employ multiple spatial GCNs, each tailored to specific feature lengths, to avoid information loss.
Datasets
The proposed method was evaluated on two datasets:
- CAD120: A relatively simpler dataset providing RGBD data, skeletal data, and pre-computed features for objects and actors. It was used for action segmentation with 10 action classes.
- Charades: A more complex, real-world dataset featuring 9848 videos with 157 action classes, 38 object classes, and 33 verb classes. It includes RGB and flow streams and presents a multi-label, multi-class problem.
Key Insights and Ideas
- Context is King for Action Segmentation: The paper strongly emphasizes that moving beyond just actor poses to incorporate rich contextual cues (scene, objects, actions) is critical for accurate and robust action segmentation, especially for subtle or complex actions.
- Flexibility is Key for Real-World Videos: The dynamic nature of real-world videos, with objects and actors appearing and disappearing, necessitates graph structures that can handle significant deformation. Flexible temporal connections and arbitrary edge connections are crucial for this.
- Hourglass Networks Adapt Well to Graphs: The successful adaptation of the hourglass architecture to graph data demonstrates its potential for improving performance in graph-based learning tasks beyond traditional grid-like data.
- Joint Inference for Better Performance: By jointly inferring across various contextual cues within a single graph network, Stacked-STGCN achieves better performance than methods that process these cues separately.
Key Results
- CAD120: Stacked-STGCN achieved an F1-score of 88.5%, outperforming the previous state-of-the-art S-RNN by approximately 5.3%.
- Charades:
- Using VGG features, Stacked-STGCN achieved an mAP of 10.94% (all features) and 19.09% (with I3D features), outperforming baselines and showing significant improvements over methods like S-RNN.
- Ablation studies showed that the hourglass structure contributed a 1.47% improvement, and flexible temporal connections (3 time steps) were optimal for Charades.
- The paper notes that their I3D-based Stacked-STGCN achieved the second-best performance on Charades, only slightly behind a method employing an attention mechanism.
Limitations and Future Work
- Limitations:
- The paper acknowledges that while their method is robust to missed detections, the performance can still be affected by subtle changes in actor appearance (as seen in an example on CAD120).
- The current edge model uses fixed weights, and exploring more sophisticated, learned weights for correlations between nodes could further improve performance.
- Future Work:
- Investigating more sophisticated edge weighting mechanisms beyond binary decisions.
- Developing methods to visualize the traces of activated nodes/edges to provide inherent explanations for action detections, as current GCN implementations do not directly support this.
- Applying Stacked-STGCN to a wider range of applications requiring structured inference over sequences with heterogeneous data types and varied temporal extents.
Summary updated