Temporal action segmentation means labeling every frame of a long video, such as a ten-minute cooking recording, with the action happening at that moment. Models for this task tend to "flicker": they switch labels back and forth between adjacent frames, which breaks one sustained action into many spurious fragments. This failure is called over-segmentation. A common fix is to supervise the network at several depths or temporal scales. That fix carries a known risk. Coarse-scale supervision usually means predicting downsampled frame labels, and those labels blur boundaries and can wipe out short actions entirely.
We propose to change what the coarse scales are asked to predict. Coarse temporal streams would no longer predict blurred frame labels. They would predict quantities that are exactly defined at their own scale: which actions occur inside a window, and how many action changes the window contains. A single full-resolution stream, kept at one token per frame throughout the network, carries the only frame-level loss. A differentiable count of predicted transitions ties the fine output to the ground-truth change count in each coarse window. In two sentences: give each temporal scale a target that is truthful at that scale, instead of a downsampled copy of the frame labels. Then penalize the fine predictions only for transitions in excess of the true number per window, which suppresses flicker without forcing real short segments to disappear.
Core Claim
Over-segmentation is a counting error before it is a smoothness error. The standard smoothing penalty, the truncated mean squared error introduced in MS-TCN, penalizes large frame-to-frame jumps in predicted log-probabilities. It does this regardless of whether a jump is a real boundary. The Gaussian-similarity variant in ASRF softens the penalty where neighboring frames look different, but it still has no notion of how many boundaries belong in a stretch of video.
Let $p_t$ be the predicted class distribution at frame $t$. For a coarse window $w$ of frames, the expected number of label changes is
$$\hat n_w = \sum_{t \in w} \left(1 - p_t^\top p_{t+1}\right),$$
which equals the exact change count when predictions are one-hot. For soft predictions it is the expected count under independent sampling of each frame's label. Let $n_w$ be the ground-truth change count in that window. We penalize
$$\mathcal{L}_{count} = \sum_w \left[\alpha\,\max(0,\hat n_w - n_w) + \beta\,\max(0, n_w - \hat n_w)\right],$$
where $\alpha \gg \beta$, so excess transitions are punished hard and missing ones only mildly. Because $n_w$ is computed exactly from the labels at every window size, this loss never contains the boundary blur that downsampled labels introduce.
The coarse streams carry two additional targets, both exact at their scale: a multi-label "actions present in this window" target and a direct regression of $n_w$. Our claim is that this scale-matched supervision reduces over-segmentation (higher Edit score and segmental F1) at no cost in short-segment recall. Conventional multi-scale deep supervision with pooled frame labels, by contrast, trades one for the other.
Where This Sits in Prior Work
The motivating evidence comes from three places.
Temporal Convolutional Networks for Action Segmentation and Detection compared an encoder-decoder network that pools time and then upsamples it against a dilated network that never pools. It found that pooling costs boundary precision. It also introduced the segmental F1 metric, which penalizes fragmentation.
MS-TCN and MS-TCN++ keep full temporal resolution and apply a loss after every refinement stage. MS-TCN++ adds layers that combine two dilation rates. Both report a plateau after four or five stages. Because every stage is supervised with the same frame-level target, added depth repeats the same signal rather than adding new information.
ASFormer keeps full resolution inside a windowed-attention Transformer and sees diminishing returns beyond nine blocks.
A constraining result comes from monocular depth. Unsupervised Scale-consistent Depth and Ego-motion Learning found that supervising at multiple downsampled resolutions over-smoothed depth edges, while native-resolution supervision alone was sharper. We read this as a warning about downsampled targets, not about multi-scale structure. Our design keeps the structure and replaces the targets.
The spatial deep-supervision literature supplies the mechanism we borrow. Stacked Hourglass Networks supervise a full heatmap after every module. Stacked Deconvolutional Networks supervise every upsampling stage. Both always use the same kind of target at every scale. Lite-HRNet supplies the architecture template: parallel high- and low-resolution streams with continuous exchange, made cheap by a channel-weighting operator. It supervises only the final output, and it has never been applied to time.
The research file found no temporal segmentation model with parallel multi-resolution streams. More importantly, it found none that supervises coarse scales with anything other than frame labels. That second absence is the novelty we stake out.
Research Program
The model operates on frozen I3D frame features, the standard setting. It has a full-resolution temporal stream built from dilated residual layers as in MS-TCN, run in parallel with streams at 1/4 and 1/16 temporal resolution. After each block the streams exchange information, using a one-dimensional adaptation of Lite-HRNet's conditional channel weighting: a pooled summary from each stream gates the channels of the others.
Supervision is split by scale. The full-resolution stream gets frame cross-entropy plus $\mathcal{L}_{count}$, computed over windows matching each coarse stream's stride. Each coarse stream gets two heads: a multi-label presence head trained with binary cross-entropy, and a change-count regressor trained toward $n_w$. Coarse heads never see frame labels.
The analysis has three parts. First, we measure how predicted-versus-true segment counts and short-segment recall move as we swap target types while holding the architecture fixed. Second, we test whether the stage plateau in MS-TCN shifts when refinement depth comes from scale-diverse supervision. Third, we sweep the asymmetry ratio $\alpha/\beta$ to map the trade-off between flicker and missed boundaries. TMSE cannot be tuned to make that trade-off explicitly.
Baselines
The naive reference point is a single-stage dilated temporal network trained with frame cross-entropy only. This is the configuration whose flicker motivated the field. It anchors how much any regularizer buys.
The architectural baseline is MS-TCN++. We run it from its standard recipe on identical I3D features and splits. Its multi-dilation layers are the strongest existing claim that multi-scale context can be obtained at full resolution without parallel streams. For MS-TCN itself the file reports 80.7% frame accuracy and 76.3 F1@10 on 50Salads (MS-TCN).
The strongest existing methods are ASFormer and DiffAct. ASFormer reports F1@10 85.1, Edit 79.6 and accuracy 85.6 on 50Salads; F1@10 76.0, Edit 75.0 and accuracy 73.5 on Breakfast; and F1@10 90.1, Edit 84.6 and accuracy 79.7 on GTEA. Diffusion Action Segmentation reports an average metric of 87.4% on 50Salads and 75.1% on Breakfast, and notes a smaller advantage on GTEA, for which no number is given in the file. DiffAct is the hardest competitor because its denoising steps act as many rounds of refinement.
For ASRF the file gives only relative gains: a 10.6% F1@50 improvement on 50Salads, and 16.1% segmental F1 and 13.7% edit improvement over MS-TCN on Breakfast. We therefore rerun ASRF ourselves rather than quote an absolute figure.
To keep the comparison fair, every method uses the same features, the standard splits, a parameter budget matched within 10%, and the same number of hyperparameter trials on held-out training videos. We report means over five seeds.
The proposal counts as beating a baseline if it improves Edit and F1@50 beyond seed variance on Breakfast and 50Salads while short-segment recall does not drop. Matching DiffAct's average with a single forward pass would itself be a meaningful result.
Three ablations isolate the key ingredient. The first keeps the identical parallel-stream architecture but supervises the coarse heads with majority-pooled frame labels; this is conventional multi-scale deep supervision. The second removes coarse supervision entirely, leaving final-output loss only, as in Lite-HRNet. The third adds $\mathcal{L}_{count}$ to plain MS-TCN++ with no parallel streams.
Why not just…?
Why not just use conventional multi-scale deep supervision with downsampled labels? This is the first ablation. The depth-estimation finding of Bian et al. predicts that pooled targets will smooth boundaries. In time, that translates to erased short actions. If pooled-label supervision matches our Edit and F1 and our short-segment recall, the scale-matched-target claim is false, and we will say so.
Why not just add the count loss to MS-TCN++ and skip the parallel streams? This is the real threat. The loss may do all the work. The third ablation settles it directly. If MS-TCN++ plus $\mathcal{L}_{count}$ is within seed variance of the full model, the contribution narrows to a loss function. That would still be useful, and simpler. The parallel streams would justify themselves only by what the presence and count heads add through feature exchange, or by shifting the stage plateau.
Why not just detect boundaries and vote, as ASRF does, or turn up the smoothing penalty? ASRF itself documents that majority voting locks in an error whenever most frames in a segment are misclassified. TMSE-style penalties have no target count, so strengthening them trades flicker for missed short segments. We test this by sweeping the TMSE weight on MS-TCN++ and plotting short-segment recall against Edit alongside our $\alpha/\beta$ sweep. The claim holds only if our curve dominates.
Proposed Experiments
We evaluate on 50Salads, Breakfast and GTEA with frozen I3D features. Following Lea et al., we report frame accuracy, Edit, and F1 at 10, 25 and 50% overlap. Two diagnostic metrics are added. One is the ratio of predicted to true segment count per video, which directly measures over-segmentation. The other is recall of ground-truth segments in the shortest duration quartile, which directly measures over-smoothing.
The decisive check is the pooled-label versus scale-matched-target comparison on the same architecture. It isolates whether the type of target, rather than the multi-scale structure, is what helps.
A second check repeats the MS-TCN stage-count sweep, from one to eight refinement steps, under our supervision. It asks whether the four-to-five-stage plateau moves.
A third check targets GTEA, the small egocentric set with heavy occlusion, where DiffAct reported its smallest advantage. There we test whether the count regressor overfits with only 28 videos. We would report that honestly as a limit of the method.
Starting code exists in the MIT-licensed SSTDA repository, which is built on MS-TCN. It also allows a follow-up asking whether count supervision reduces the over-segmentation that SSTDA attributes to shifts in appearance between subjects.
References
- MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation
- Alleviating Over-segmentation Errors by Detecting Action Boundaries
- Temporal Convolutional Networks for Action Segmentation and Detection
- MS-TCN++: Multi-Stage Temporal Convolutional Network for Action Segmentation
- ASFormer: Transformer for Action Segmentation
- Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video
- Stacked Hourglass Networks for Human Pose Estimation
- Stacked Deconvolutional Network for Semantic Segmentation
- Lite-HRNet: A Lightweight High-Resolution Network
- Diffusion Action Segmentation
- Action Segmentation With Joint Self-Supervised Temporal Domain Adaptation