Abstract
We propose an effective framework for the temporal action segmentation task, namely an Action Segment Refinement Framework (ASRF). Our model architecture consists of a long-term feature extractor and two branches: the Action Segmentation Branch (ASB) and the Boundary Regression Branch (BRB). The long-term feature extractor provides shared features for the two branches with a wide temporal receptive field. The ASB classifies video frames with action classes, while the BRB regresses the action boundary probabilities. The action boundaries predicted by the BRB refine the output from the ASB, which results in a significant performance improvement. Our framework refines frame-level hypotheses of action classes using predicted action boundaries and utilizes a novel loss function for smoothing the transition of action probabilities. ASRF outperforms state-of-the-art methods on three challenging datasets, offering an improvement of up to 13.7% in segmental edit distance and up to 16.1% in segmental F1 score.
Introduction
In temporal action segmentation, the objective is to assign an action label (e.g., "cutting tomato," "peeling cucumber") to every frame in a long, untrimmed video. While deep learning has improved frame-wise accuracy, models frequently suffer from over-segmentation. This "flickering" problem occurs when a model's predicted labels flip rapidly between classes within a single continuous action, resulting in a fragmented and unrealistic timeline. Most existing models struggle because they focus on individual frame classification without a strong grasp of the global temporal structure or the precise moments when actions transition.
The Action Segment Refinement Framework (ASRF)
To address over-segmentation, we introduce the Action Segment Refinement Framework (ASRF). The core philosophy of ASRF is to decouple action classification from boundary detection, allowing the model to determine what is happening separately from when it changes.
Model Architecture
The framework is composed of three primary components:
- Long-term Feature Extractor: A Temporal Convolutional Network (TCN) with dilated residual layers that captures a wide temporal receptive field, providing rich contextual features to both subsequent branches.
- Action Segmentation Branch (ASB): This branch performs frame-wise classification, predicting the probability of each action class for every frame.
- Boundary Regression Branch (BRB): This novel, class-agnostic branch predicts the probability that a frame represents a boundary between any two actions. By focusing solely on transitions, it learns structural patterns that classification-only heads often miss.
Refinement Strategy and Loss Functions
During the inference phase, the ASRF uses the peaks identified by the BRB to partition the video into discrete segments. For each segment, the model aggregates the frame-wise predictions from the ASB and performs a majority vote. Every frame within that segment is then reassigned to the winning class, effectively smoothing out the "salt-and-pepper" noise of over-segmentation.
To further improve performance, we propose the Gaussian Similarity-weighted Truncated Mean Squared Error (GS-TMSE) loss. Unlike standard smoothing losses that penalize all probability changes equally, GS-TMSE uses a Gaussian kernel to evaluate the visual similarity between adjacent frames. If frames are visually distinct, the model is penalized less for changing its prediction, allowing for sharper and more accurate transitions at true action boundaries.
Experimental Results
We evaluated ASRF on three benchmark datasets: 50 Salads, GTEA, and the Breakfast Dataset. The framework demonstrated substantial improvements over previous state-of-the-art models, particularly in metrics that measure segmentation quality rather than just frame-wise accuracy.
Key performance gains include:
- Segmental F1 Score: Improved by up to 16.1% on the Breakfast dataset.
- Segmental Edit Distance: Improved by up to 13.7%.
- Qualitative Improvement: As shown in the visualizations, the refinement step transforms a noisy, fragmented prediction into a stable timeline that closely mirrors human-annotated ground truth.
Conclusion
The ASRF provides a robust solution to the over-segmentation problem by treating boundary detection as a first-class task. By decoupling "what" from "when" and employing a majority-voting refinement strategy, we achieve significant gains in temporal consistency. While the framework is a powerful refinement tool, its success remains linked to the underlying quality of the Action Segmentation Branch. Future research will explore tighter integration between classification and boundary confidence to further enhance the reliability of the refinement process.
Summary updated