TL;DR
This paper introduces a novel framework, "Stitch, Contrast, and Segment" (SCS), that enables learning human action segmentation models from readily available trimmed skeleton videos, which can then effectively segment actions in untrimmed videos. This approach tackles the significant challenge of data annotation for real-world action segmentation by leveraging a clever stitching mechanism and a contrastive learning strategy.
What Problem Does This Paper Address?
Human action segmentation, the task of identifying the start and end times of specific actions within a continuous video stream, is crucial for many applications like surveillance, robotics, and human-computer interaction. Traditional methods for training action segmentation models rely heavily on untrimmed videos with precise frame-level annotations, which are incredibly time-consuming and expensive to create. This paper addresses the critical gap: how to train robust action segmentation models when only abundant, but trimmed (single-action), skeleton videos are available, and the target application involves untrimmed, multi-action videos. The core challenge is bridging the temporal granularity difference between training data (short, single actions) and testing data (long, multiple actions).
Methods
The proposed framework, "Stitch, Contrast, and Segment" (SCS), operates in three key stages:
- Stitch (Skeleton-Stitch): This novel technique addresses the data scarcity of untrimmed videos by creating synthetic multi-action sequences. It takes trimmed skeleton videos and, using frame correspondence (aligning joint positions), "stitches" them together to form longer, plausible sequences that mimic untrimmed videos. This allows the model to learn how different actions connect temporally.
- Contrast (Granular Contrastive Learning - GCL): To learn meaningful representations from these stitched sequences, the paper introduces GCL. Unlike previous contrastive methods that might focus on individual actions, GCL is designed for action segmentation. It learns to distinguish between different action categories within the stitched sequences, aiming to capture fine-grained temporal embeddings that are crucial for segmenting actions at the frame level. This involves a skeleton encoder (based on ST-GCN) and a projection layer, trained with a novel pretext task.
- Segment: Finally, a segmentation layer (implemented using a Temporal Convolutional Network - TCN) is trained on top of the learned skeleton encoder. This layer takes the frame-wise representations and predicts action labels for each frame, effectively performing the action segmentation. The framework supports both supervised adaptation (fine-tuning with some labeled untrimmed data) and zero-shot adaptation (no labeled untrimmed data).
Datasets
The paper utilizes several standard datasets for training and evaluation:
- NTU RGB+D: A large-scale 3D skeleton dataset with 56,880 trimmed sequences covering 60 daily actions. This serves as a primary source domain.
- PKU-MMD v1 & v2: Popular 3D untrimmed skeleton datasets. PKU-MMD v1 has 21,545 class-specific sequences when trimmed, while v2 contains 1,009 untrimmed videos with 7,000 class-specific sequences. These serve as target domains.
- Toyota Smarthome (TST) & Toyota Smarthome Untrimmed (TSU): 2D datasets for daily living activities, used to evaluate the method on different modalities (2D vs. 3D) and domains.
Key Insights and Ideas
- Bridging the Trimmed-to-Untrimmed Gap: The core insight is that by intelligently "stitching" trimmed videos, we can create synthetic untrimmed data that effectively trains action segmentation models, bypassing the need for expensive manual annotation of real untrimmed videos.
- Frame Correspondence for Stitching: The use of frame correspondence (aligning joint positions of consecutive frames from different sequences) is a clever way to ensure smooth temporal transitions when stitching actions, making the synthetic data more realistic.
- Granular Contrastive Learning: The development of GCL specifically for action segmentation, rather than general action recognition, is key. It focuses on learning representations that capture temporal context and subtle action boundaries, which are essential for frame-level prediction.
- Temporal Domain Adaptation: The framework explicitly frames the problem as temporal domain adaptation, transferring knowledge from a domain of coarse temporal granularity (trimmed videos) to a domain of fine temporal granularity (untrimmed videos).
Key Results
The proposed SCS framework demonstrates significant improvements across various temporal domain adaptation scenarios:
- Supervised Adaptation: SCS (s.a.) consistently outperforms state-of-the-art methods that rely on sliding windows for adaptation, achieving substantially higher mIoU and Acc scores on target datasets like PKU-MMD v1 and v2. For instance, on NTU to PKU2, SCS (s.a.) achieved 53.4% mIoU and 69.5% Acc, significantly outperforming baselines.
- Zero-Shot Adaptation: SCS (z.a.) shows the first successful attempt at trimmed-to-untrimmed zero-shot learning for action segmentation, achieving competitive results even without any target domain labels. For example, in the PKU1 to PKU2 zero-shot setting, SCS (z.a.) achieved 24.9% mIoU and 36.7% Acc, outperforming baselines that used original trimmed sequences.
- Effectiveness of Stitching: Ablation studies show that using stitched sequences (Dataset II) for training leads to comparable performance to using original untrimmed datasets, whereas simple concatenation without frame correspondence (Dataset I) drastically degrades performance.
Limitations and Future Work
- Data Augmentation: While the paper shows that combining original data with stitched data (Expanded Dataset) improves performance, it also notes that classical data augmentation strategies can be orthogonal and complementary to their method.
- Cross-Sequence Generalization: The paper highlights that models trained on original datasets can exhibit poor generalization on unseen action permutations (Cross-Sequence protocol). However, training with the expanded, stitched datasets improves this generalization.
- Future Work: The authors suggest that their framework can be extended to handle more complex scenarios and potentially other modalities. The modularity of the SCS framework allows for exploring different learning strategies and adaptation objectives in the target domain. The availability of the code and supplementary material is also noted.
Summary updated