Temporal action segmentation (TAS) is critical for video understanding but traditionally requires labor-intensive frame-by-frame annotations. Timestamp-supervised TAS (TSTAS) offers a cost-effective alternative by using only one labeled frame per action. However, TSTAS often suffers from label bias, where models over-fit to specific frames or fail to capture the full semantic range of complex actions. This paper introduces D-TSTAS, a framework designed to mitigate these biases through Masked Timestamp Predictions (MTP) and Center-oriented Timestamp Expansion (CTE). D-TSTAS achieves state-of-the-art performance on major benchmarks, significantly narrowing the gap between timestamp-supervised and fully supervised methods.
The Problem of Label Bias
Temporal Action Segmentation involves identifying the start, end, and category of every action in long, untrimmed videos (e.g., "pouring cereal," "adding milk"). While full supervision yields high accuracy, the cost of labeling every frame is prohibitive. TSTAS reduces this burden by requiring only a single click per action. Despite its efficiency, TSTAS introduces two primary forms of label bias:
- Focus Bias: Models tend to recognize only frames visually similar to the single annotated timestamp, often missing the start and end of actions if they look different.
- Representation Bias: Complex actions involve different visual stages (e.g., reaching for a knife vs. the actual slicing). A single timestamp cannot represent this entire semantic range.
The D-TSTAS Framework
The proposed Debiasing-TSTAS (D-TSTAS) framework addresses these challenges in two distinct phases:
1. Initialization via Masked Timestamp Predictions (MTP)
To prevent the model from over-relying on the annotated frame, D-TSTAS masks the features of the annotated frame during training. This forces the model to leverage surrounding unannotated frames to predict the action label, thereby learning broader contextual information from the outset. This technique, inspired by self-supervised learning, ensures the model understands the "why" behind an action's context rather than just memorizing a single frame.
2. Refinement via Center-oriented Timestamp Expansion (CTE)
Following initialization, the CTE module iteratively expands the single annotated point into a set of "pseudo-timestamps." By identifying the center of predicted segments and adding semantically distinct frames as new anchors, CTE builds a more robust, "semantic-rich" representation of the entire action. This effectively turns one label into several high-quality pseudo-labels.
3. Segmental Confidence Loss
Standard models often use a confidence loss that assumes certainty peaks at the center and drops at the edges, which can blur action boundaries. D-TSTAS introduces a loss that maintains high, flat confidence across the entire predicted segment and only penalizes changes at the edges, leading to much sharper and more precise boundaries.
Experimental Results
D-TSTAS was evaluated on three standard benchmarks: 50Salads, Breakfast, and GTEA. The framework set new state-of-the-art benchmarks for timestamp-supervised methods:
- Superior Accuracy: On the GTEA dataset, D-TSTAS achieved an F1@10 score of 91.5%, significantly outperforming previous leading timestamp methods (e.g., 78.9% by Li et al.).
- Competitive with Full Labels: D-TSTAS performs within 1-3% of models trained on 100% frame labels across several metrics, despite using only a tiny fraction of the annotation effort.
- Boundary Precision: Qualitative results show that D-TSTAS is much better at identifying the exact moment an action starts and ends, avoiding the common "over-segmentation" errors where one action is chopped into many tiny pieces.
Conclusion and Future Work
D-TSTAS effectively addresses label bias in timestamp-supervised temporal action segmentation. By forcing context learning through masking and iteratively expanding semantic anchors, the framework provides a highly efficient yet accurate solution for video understanding. A primary remaining limitation is the reliance on at least one timestamp per action; if an annotator misses a segment, the model currently lacks a mechanism to discover it. Future work may involve integrating unsupervised discovery techniques to handle such inconsistencies in human annotation.
Summary updated