Abstract
Temporal action segmentation often suffers from over-segmentation errors, where continuous actions are incorrectly fragmented into multiple short segments. We present the Action Segment Refinement Framework (ASRF), which decouples action classification from boundary detection. By employing a dedicated Boundary Regression Branch (BRB) alongside a traditional Action Segmentation Branch (ASB), ASRF identifies precise action transitions and refines frame-level predictions through a segment-wise majority voting scheme. Evaluated on the Breakfast, 50 Salads, and GTEA datasets, ASRF demonstrates significant improvements, including a 16.1% boost in segmental F1 score on the Breakfast dataset.
Introduction
In the domain of video analysis, temporal action segmentation aims to assign a semantic label to every frame in an untrimmed video. While Temporal Convolutional Networks (TCNs) have become the standard for this task, they frequently produce "jittery" or over-segmented outputs. These errors manifest as rapid, erroneous label switching within a single, continuous action, often referred to as the "salt-and-pepper" effect. Although frame-level accuracy might remain high, the lack of temporal consistency severely degrades the utility of these models for real-world applications.
Methodology: The Action Segment Refinement Framework (ASRF)
The ASRF architecture addresses over-segmentation by separating the identification of action boundaries from the classification of the actions themselves.
Core Components
- Long-term Feature Extractor: A shared backbone utilizing dilated residual layers to capture broad temporal context across the video sequence.
- Action Segmentation Branch (ASB): A standard classification branch that predicts action labels for each frame.
- Boundary Regression Branch (BRB): A specialized branch that outputs a probability score for action transitions. It focuses solely on detecting boundaries, regardless of the specific action classes involved.
Refinement via Majority Voting
During inference, the framework identifies local peaks in the BRB output to partition the video into discrete segments. For each segment, ASRF performs a majority vote based on the ASB's frame-level predictions. This process effectively filters out transient classification noise, ensuring that each detected segment is assigned a single, consistent action label.
Gaussian Similarity-weighted TMSE (GS-TMSE)
To further improve temporal consistency during training, the authors introduce the GS-TMSE loss. Unlike standard smoothing losses that penalize all label transitions equally, GS-TMSE uses a Gaussian kernel to weight penalties based on the visual similarity of adjacent frames. This approach preserves sharp transitions at actual boundaries while aggressively smoothing noise in visually similar regions.
Experiments and Results
The framework was evaluated on three benchmark datasets: 50 Salads (17 classes), GTEA (7 classes), and the Breakfast Dataset (77 hours of video).
Quantitative Performance
ASRF achieved state-of-the-art results across all benchmarks. Notably, on the Breakfast dataset, it realized a 16.1% improvement in segmental F1 score and a 13.7% improvement in edit distance over the previous baseline (MS-TCN). On the 50 Salads dataset, the F1@50 score—a rigorous metric for temporal overlap—improved by 10.6%.
Qualitative Analysis
Visualizations of the model's output confirm the effectiveness of the refinement process. As shown in the qualitative comparisons, ASRF eliminates the fragmented predictions typical of ASB-only models, producing clean segments that align closely with human-annotated ground truth.
Discussion and Conclusion
The success of ASRF highlights the importance of decoupling boundary detection from action classification. By treating boundaries as class-agnostic features, the model becomes more robust to the "jitter" inherent in frame-by-frame classification. The majority voting mechanism serves as a powerful temporal filter, though it remains dependent on the underlying accuracy of the ASB; if the classifier consistently misidentifies an entire segment, the refinement step will solidify that error.
In conclusion, ASRF provides a robust solution to the over-segmentation problem in temporal action segmentation. By integrating boundary-aware refinement and similarity-weighted smoothing, the framework achieves significant performance gains and produces more semantically coherent action sequences.
Summary updated