When deep learning models analyze long, untrimmed videos to perform temporal action segmentation, they frequently encounter a fundamental failure mode: temporal over-segmentation, also known as label flickering or prediction jitter. Instead of outputting a clean, continuous block of time for a single activity (such as "cooking an egg" or "performing a surgical gesture"), the model's predictions rapidly toggle between different action classes frame-by-frame.
As observed by Colin Lea et al. in Temporal Convolutional Networks for Action Segmentation and Detection, this lack of temporal consistency results in highly fragmented outputs that do not correspond to the actual duration or physical reality of human actions. Understanding why this flickering happens—and how modern architectures resolve it—is key to building robust video understanding systems.
The Root Cause: Local Noise vs. Global Context
To understand why models over-segment, consider how a standard frame-level classifier operates. When a model processes a video, it typically extracts visual features from individual frames or very short clips. If it is trained using standard cross-entropy loss, the optimization treats each frame as an independent classification task.
However, human activities are rarely uniform. A single continuous action often contains internal variations, such as brief pauses, changes in hand configuration, or subtle shifts in motion. For instance, Low Jian He et al. demonstrate in Hands-On: Segmenting Individual Signs from Continuous Sequences that models frequently mistake internal pauses within a single long sign-language gesture as the start of a completely new sign. Because the frame-level classifier prioritizes local visual features over global context, any momentary drop in discriminability or minor visual noise triggers an unrealistic, rapid transition to a different label. As Yazan Abu Farha et al. point out in [When Will You Do What?
- Anticipating Temporal Occurrences of Activities](/work/when-will-you-do-what-anticipating-temporal-occurrences), standard classification losses lack the inherent smoothing constraints required to represent the sustained duration of real-world behaviors.
To solve this problem, researchers have developed three primary strategies: multi-stage refinement, boundary-decoupled classification, and explicit duration constraints.
Strategy 1: Multi-Stage Refinement and Smoothing Losses
One of the most influential insights in combating over-segmentation is that action classification and temporal smoothing should be handled progressively. Rather than trying to generate a perfect sequence in a single pass, models can generate a rough initial prediction and then iteratively clean up the noise.
This is the core intuition behind the Multi-Stage Temporal Convolutional Network (MS-TCN). In MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation, Juergen Gall et al. introduced an architecture that stacks multiple dilated convolutional stages. The first stage operates directly on the video features to produce an initial, often jittery, frame-level classification. Subsequent stages then take the probability distributions of the previous stage as input, rather than the raw features, and apply dilated convolutions with a large receptive field to smooth out the predictions.
[ Video Frames ] ---> [ Stage 1: Initial TCN ] ---> [ Jittery Predictions ]
|
v
[ Stage 2: Dilated TCN ] ---> [ Smoothed Predictions ]
|
v
[ Stage 3: Dilated TCN ] ---> [ Final Consistent Output ]
To force these higher stages to eliminate flickering, the authors introduced a specialized truncated mean squared error (TMSE) loss. This loss function calculates the difference between the predicted log-probabilities of consecutive frames. If the model predicts a sudden change in class from frame $t$ to frame $t+1$, the smoothing loss penalizes the network heavily unless that change is sustained. By combining multi-stage dilated convolutions with this temporal penalty, the network learns to ignore short-lived visual fluctuations. This paradigm was further optimized by Shijie Li et al. in MS-TCN++: Multi-Stage Temporal Convolutional Network for Action Segmentation through the use of dual dilated layers, which capture multi-scale temporal context more efficiently to prevent continuous actions from being fragmented.
Strategy 2: Decoupling Classification from Boundary Detection
A second intuitive approach is to separate the task of determining what action is occurring from where the action begins and ends. When a model is forced to perform both tasks simultaneously on a frame-by-frame basis, boundary frames become highly unstable, leading to localized flickering.
The Action Segment Refinement Framework (ASRF), proposed by Yuchi Ishikawa et al. in Alleviating Over-segmentation Errors by Detecting Action Boundaries, addresses this by creating two distinct branches in the network: an action classification branch and a dedicated action boundary detection branch. The boundary branch is trained specifically to detect the precise transitions between actions. Once the boundaries are established, the framework applies a "voting" mechanism within each bounded segment. Because the model knows exactly where a segment starts and ends, it can aggregate all frame-level predictions within those boundaries and assign a single, consensus label to the entire block, completely neutralizing internal frame-level jitter.
+---> [ Action Classification Branch ] ---\
| v
[ Video Features ] ---+ [ Boundary Voting ] ---> [ Stable Segments ]
| ^
+---> [ Boundary Detection Branch ] ------/
Shifting from dense frame-wise classification to a sparse, segment-centric view represents the latest evolution of this concept. In Efficient Temporal Action Segmentation via Boundary-aware Query Voting, Peiyao Wang et al. developed a Transformer-based framework (BaFormer) that completely decouples action masks from their semantic classes using a query-based strategy. By predicting a fixed set of action segments directly rather than labeling every individual frame, the model inherently avoids the possibility of frame-level flickering, while simultaneously reducing inference time and computational overhead.
Strategy 3: Enforcing Explicit Duration and Context Constraints
Over-segmentation is particularly severe when models must operate in real-time (online settings) or adapt to unfamiliar environments, as they cannot look ahead to verify if a label change is permanent.
In online scenarios, where future frames are unavailable, models are highly vulnerable to local noise. To stabilize real-time outputs, Qing Zhong et al. introduced a dedicated online baseline in OnlineTAS: An Online Baseline for Temporal Action Segmentation. They mitigated real-time jitter by implementing a confidence-based smoothing heuristic that enforces a minimum expected duration for any predicted action state. If the model attempts to switch labels due to a momentary visual anomaly, the heuristic suppresses the transition unless the new prediction maintains high confidence over a logical window of time.
Similarly, when models encounter domain shifts (such as different people performing the same task at varying speeds), local feature representations can degrade, causing predictions to fragment. The authors of Action Segmentation With Joint Self-Supervised Temporal Domain Adaptation resolved this by introducing self-supervised auxiliary tasks designed to align global temporal dynamics across domains. By forcing the network to learn the overarching, macro-level flow of an activity rather than over-indexing on fine-grained, actor-specific visual details, the model naturally maintains a stable, continuous prediction signal even when faced with novel environments.