Alleviating Over-segmentation Errors by Detecting Action Boundaries

Yuchi Ishikawa, Seito Kasai, Yoshimitsu Aoki, Hirokatsu Kataoka

Abstract

We propose an effective framework for the temporal action segmentation task, namely an Action Segment Refinement Framework (ASRF). Our model architecture consists of a long-term feature extractor and two branches: the Action Segmentation Branch (ASB) and the Boundary Regression Branch (BRB). The long-term feature extractor provides shared features for the two branches with a wide temporal receptive field. The ASB classifies video frames with action classes, while the BRB regresses the action boundary probabilities. The action boundaries predicted by the BRB refine the output from the ASB, which results in a significant performance improvement. Our framework refines frame-level hypotheses of action classes using predicted action boundaries and utilizes a novel loss function for smoothing the transition of action probabilities. ASRF outperforms state-of-the-art methods on three challenging datasets, offering an improvement of up to 13.7% in segmental edit distance and up to 16.1% in segmental F1 score.

Introduction

In temporal action segmentation, the objective is to assign an action label (e.g., "cutting tomato," "peeling cucumber") to every frame in a long, untrimmed video. While deep learning has improved frame-wise accuracy, models frequently suffer from over-segmentation. This "flickering" problem occurs when a model's predicted labels flip rapidly between classes within a single continuous action, resulting in a fragmented and unrealistic timeline. Most existing models struggle because they focus on individual frame classification without a strong grasp of the global temporal structure or the precise moments when actions transition.

The Action Segment Refinement Framework (ASRF)

To address over-segmentation, we introduce the Action Segment Refinement Framework (ASRF). The core philosophy of ASRF is to decouple action classification from boundary detection, allowing the model to determine what is happening separately from when it changes.

The ASRF architecture: A long-term feature extractor feeds into the Action Segmentation Branch (ASB) and the Boundary Regression Branch (BRB). The BRB's boundary predictions are used to refine the ASB's initial over-segmented output.

Model Architecture

The framework is composed of three primary components:

  1. Long-term Feature Extractor: A Temporal Convolutional Network (TCN) with dilated residual layers that captures a wide temporal receptive field, providing rich contextual features to both subsequent branches.
  2. Action Segmentation Branch (ASB): This branch performs frame-wise classification, predicting the probability of each action class for every frame.
  3. Boundary Regression Branch (BRB): This novel, class-agnostic branch predicts the probability that a frame represents a boundary between any two actions. By focusing solely on transitions, it learns structural patterns that classification-only heads often miss.

Refinement Strategy and Loss Functions

During the inference phase, the ASRF uses the peaks identified by the BRB to partition the video into discrete segments. For each segment, the model aggregates the frame-wise predictions from the ASB and performs a majority vote. Every frame within that segment is then reassigned to the winning class, effectively smoothing out the "salt-and-pepper" noise of over-segmentation.

To further improve performance, we propose the Gaussian Similarity-weighted Truncated Mean Squared Error (GS-TMSE) loss. Unlike standard smoothing losses that penalize all probability changes equally, GS-TMSE uses a Gaussian kernel to evaluate the visual similarity between adjacent frames. If frames are visually distinct, the model is penalized less for changing its prediction, allowing for sharper and more accurate transitions at true action boundaries.

Experimental Results

We evaluated ASRF on three benchmark datasets: 50 Salads, GTEA, and the Breakfast Dataset. The framework demonstrated substantial improvements over previous state-of-the-art models, particularly in metrics that measure segmentation quality rather than just frame-wise accuracy.

Qualitative results across four datasets (a-d). ASRF significantly reduces the 'flickering' noise seen in the ASB-only predictions, bringing the final timeline much closer to the Ground Truth (G.T.).

Key performance gains include:

  • Segmental F1 Score: Improved by up to 16.1% on the Breakfast dataset.
  • Segmental Edit Distance: Improved by up to 13.7%.
  • Qualitative Improvement: As shown in the visualizations, the refinement step transforms a noisy, fragmented prediction into a stable timeline that closely mirrors human-annotated ground truth.

Conclusion

The ASRF provides a robust solution to the over-segmentation problem by treating boundary detection as a first-class task. By decoupling "what" from "when" and employing a majority-voting refinement strategy, we achieve significant gains in temporal consistency. While the framework is a powerful refinement tool, its success remains linked to the underlying quality of the Action Segmentation Branch. Future research will explore tighter integration between classification and boundary confidence to further enhance the reliability of the refinement process.

Summary updated

Abstract

We propose an effective framework for the temporal action segmentation task, namely an Action Segment Refinement Framework (ASRF). Our model architecture consists of a long-term feature extractor and two branches: the Action Segmentation Branch (ASB) and the Boundary Regression Branch (BRB). The long-term feature extractor provides shared features for the two branches with a wide temporal receptive field. The ASB classifies video frames with action classes, while the BRB regresses the action boundary probabilities. The action boundaries predicted by the BRB refine the output from the ASB, which results in a significant performance improvement. Our contributions are three-fold: (i) We propose a framework for temporal action segmentation, the ASRF, which divides temporal action segmentation into frame-wise action classification and action boundary regression. Our framework refines frame-level hypotheses of action classes using predicted action boundaries. (ii) We propose a loss function for smoothing the transition of action probabilities, and analyze combinations of various loss functions for temporal action segmentation. (iii) Our framework outperforms state-of-the-art methods on three challenging datasets, offering an improvement of up to 13.7% in terms of segmental edit distance and up to 16.1% in terms of segmental F1 score. Our code will be publicly available soon.
BibTeX
@inproceedings{ishikawa2021alleviating,
  author = {Yuchi Ishikawa and Seito Kasai and Yoshimitsu Aoki and Hirokatsu Kataoka},
  title = {{Alleviating Over-segmentation Errors by Detecting Action Boundaries}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  doi = {10.1109/wacv48630.2021.00237},
}
Figure