Many strong prediction systems work like a writer and a series of editors. A first network produces a rough prediction, and each later network takes that prediction as input and corrects it. Temporal action segmentation, the task of labeling every video frame with the action being performed, is the textbook case: MS-TCN stacks temporal convolution stages that each read the previous stage's class probabilities. Accuracy improves for four or five stages and then stops. The usual explanation is that refinement simply runs out of things to fix. This note argues for a different explanation that can be tested. When stages are trained end to end, every stage's weights are pulled by two forces. The first is its own loss, which asks for a confident, correct answer now. The second is the losses of all later stages, which reach back through the prediction the stage passed forward and ask for a draft that is easy to edit. If these two pulls disagree, adding stages adds more disagreement rather than more correction.
The proposal measures this disagreement directly, stage by stage, during training. It then tests a cheap correction: combine the two pulls with a conflict-averse rule instead of simply adding them. The research gives equal weight to diagnosis and to method. If a trivial fix works just as well, such as cutting the gradient between stages or reweighting stages with a fixed schedule, the experiments are designed to show that clearly.
Core Claim
Work on multi-objective loss balancing treats objectives as a flat set of parallel heads on shared weights. GradNorm equalizes gradient magnitudes across tasks. CAGrad chooses an update that avoids hurting any task while staying close to the average gradient. RECON finds the layers where task gradients conflict and gives each task its own copy of those layers. Stacked refinement has a structure these methods ignore: stage $s$'s output is stage $s{+}1$'s input, so conflict can arise along the forward chain as well as in shared weights.
Let $\theta_s$ be stage $s$'s parameters, $\hat y_s$ its output, and $\mathcal{L}_t$ the loss of stage $t$. The gradient on stage $s$ splits exactly into an "own" part and a "downstream" part:
$$ g_s \;=\; \underbrace{\nabla_{\theta_s}\mathcal{L}_s}_{\text{own}} \;+\; \underbrace{\sum_{t>s}\nabla_{\theta_s}\mathcal{L}_t}_{\text{downstream, through } \hat y_s}. $$
The claim has three parts. First, the cosine between these two parts turns increasingly negative as stages are added. Second, this negative cosine predicts where the plateau begins. Third, combining the two parts with a two-objective conflict-averse rule, applied separately at each stage, moves the plateau to more stages and reduces the rate at which later stages overwrite frames that earlier stages had right. The cost is one extra backward pass per stage, which grows linearly with depth and not with the number of stage pairs. What is new is treating "being right now" and "being useful to the next editor" as the two objectives to be balanced. Prior work balances all stage losses as peers, which is exactly what MS-TCN's unweighted sum already does.
Relation to Prior Work
Two lines of evidence motivate the proposal. MS-TCN and MS-TCN++ supervise every stage with the same classification loss plus a smoothing loss weighted by a fixed $\lambda$. They sum these losses equally across stages, share refinement weights in the ++ variant, and report both the saturation after a few stages and loss tuning as unresolved. Zero Time Waste shows the same symptom in early-exit classifiers: on VGG16, over 60% of intermediate-classifier mistakes were on inputs an earlier classifier had gotten right. We borrow this "hindsight" measurement and apply it across refinement stages.
Two other papers constrain the design. Deep Trajectory Supervision finds that forcing early layers of a single deep network toward hard terminal targets fights the natural build-up of certainty. It fixes this with a hand-set exponential softening schedule and was only tested on vertical backbones. That makes it the most direct rival explanation: if early stages merely need softer targets, a fixed schedule suffices and our gradient analysis adds nothing. Deep Improvement Supervision supervises every step of a weight-shared recursive model with targets built by corrupting the answer. It weights steps uniformly and does not measure interference between steps. Multi-step loss for reflection removal accumulates losses over recursive passes, also with uniform weights.
The method is supplied by CAGrad, which gives the two-objective combination rule. RECON contributes both the layer-wise conflict scoring we use for diagnosis and a warning we take seriously: projection methods change the direction of each update but do not reduce how often conflicts occur.
Research Program
Diagnosis. We first train MS-TCN and MS-TCN++ with 2 to 12 stages. During training we log three quantities for every stage and layer: the own-versus-downstream cosine, the ratio of their norms, and a per-stage "overwrite rate." The overwrite rate is the fraction of frames that stage $s{-}1$ labels correctly but stage $s$ labels wrongly. In the parameter-shared MS-TCN++ refinement block, we also record the conflict between stage losses in the shared weights, using RECON-style layer scores. This separates two possible sources of conflict: the forward chain and the shared weights.
Within each stage we also measure the cosine between the classification-loss gradient and the smoothing-loss gradient. A per-stage sweep of $\lambda$ then asks whether early and late stages want different smoothing strength. We keep this as a secondary question.
Intervention. At each stage, we replace the sum $g_s$ with the CAGrad combination of its two parts. Computationally, the own part comes from a backward pass in which $\hat y_s$ is detached before it enters the next stage, and the downstream part is the full gradient minus the own part. For shared weights we apply the same rule to the summed own and downstream parts.
We then test whether the effect carries over to a second domain. We apply the method to the recursive reflection-removal pipeline, where weights are shared across passes and the "stages" are recursion steps.
Baselines
The naive reference is MS-TCN and MS-TCN++ exactly as published: an unweighted sum over stages with fixed $\lambda$. It is the reference because it is what practitioners use. On 50Salads, MS-TCN reports 80.7% frame accuracy and F1@10 of 76.3 (MS-TCN). MS-TCN++ reports an F1@{10,25,50} figure of 70.1 against 64.5 for MS-TCN (MS-TCN++), and this is the strongest reported number in the file. No reported result was found on Lacuna for GTEA or Breakfast in numeric form, so for those datasets we reproduce both models under our own pipeline and compare against the reproductions.
The strongest balancing competitors are off-the-shelf GradNorm and CAGrad, each treating the $S$ stage losses as $S$ parallel tasks. They represent the obvious transfer from multi-task learning, and CAGrad reports the best NYUv2 result in the file, a 0.20% per-task performance drop against 3.97% for PCGrad (CAGrad). A DTS-style exponential softening of early-stage targets is the strongest schedule-based competitor.
Two ablations isolate the key ingredient. The first is a stop-gradient model that deletes the downstream part entirely. The second applies the conflict-averse rule to shared-weight conflicts only, leaving the chain untouched.
Fairness rules apply to every method. All use the same pre-extracted video features, splits, and epochs, and every method receives the same budget of 20 tuning runs. For our method that budget covers CAGrad's radius $c$. For GradNorm it covers the asymmetry parameter. For DTS it covers the two schedule parameters. For the static baselines it covers per-stage weights and $\lambda$. We report means over five seeds.
Beating the baselines means two things. At the best stage count, the method must improve F1@50 and edit score over the tuned static schedule by more than seed variance. It must also keep improving past the stage count at which the naive model plateaus.
Why not just…?
Why not just stop the gradient between stages? This is the most serious threat. If the downstream part is mostly harmful, deleting it is free and matches our method. Stacked Deconvolutional Networks report that joint training helps because earlier units adapt to what later units need, which suggests the downstream signal carries value. That claim was made about segmentation units, not about temporal stages. Our stop-gradient ablation settles the question directly. If it ties our method, we will report the plateau as a chain-conflict effect whose remedy is detachment, and drop the balancing machinery.
Why not just use a fixed decaying stage weight or DTS softening? A fixed schedule changes how much each stage's loss counts. It cannot change the sign of the conflict between a stage's own and downstream gradients. DTS itself notes that its schedule needs hand-tuned parameters and may not transfer across network designs. This is still a real threat. If the tuned schedule matches our method at equal tuning budget, the gradient analysis is useful as explanation but unnecessary as a method, and we will say so.
Why not just run GradNorm or CAGrad over the stage losses? These methods treat stages as peers. They therefore cannot tell the part of a later loss that flows through $\hat y_s$ apart from the part that acts on shared weights. RECON further shows that projection alone does not reduce how often conflicts occur. The ablation that applies CAGrad at the stage level versus the own/downstream level answers this question directly.
Why not just untie the refinement weights? MS-TCN has untied stages and still plateaus. That suggests shared weights are not the whole story, but it does not rule them out as a contributor. Comparing MS-TCN with MS-TCN++ under identical logging separates the two sources of conflict.
Proposed Experiments and Decisive Checks
The primary benchmarks are 50Salads, GTEA, and Breakfast, evaluated on frame accuracy, F1@{10,25,50}, edit score, and the per-stage overwrite rate. The transfer test uses the SIR² benchmark with the Nature data, following the reflection-removal paper. That paper reports 25.18 PSNR and 0.904 SSIM, which serve as the reference we must reproduce before intervening.
The first decisive check is correlational. Across stage counts and seeds, the stage at which the own/downstream cosine first turns reliably negative should predict the stage at which accuracy stops improving. If the plateau appears while the cosines are still positive, the hypothesis is wrong.
The second check is causal. At 8 to 12 stages, our method should continue to improve accuracy, reduce overwrite rate, and beat both stop-gradient and the tuned schedule. If stop-gradient or the schedule matches it, the balancing idea is refuted in favor of that simpler fix.
The third check is whether the result holds for shared weights. In the recursive reflection pipeline, the method should reduce late-pass artifacts, measured in PSNR and LPIPS, at matched runtime. If the gain appears in action segmentation but not here, the effect is specific to untied chains and not a general property of refinement.
References
- MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation
- GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks
- Conflict-Averse Gradient Descent for Multi-task learning
- Recon: Reducing Conflicting Gradients From the Root For Multi-Task Learning
- MS-TCN++: Multi-Stage Temporal Convolutional Network for Action Segmentation
- Zero Time Waste: Recycling Predictions in Early Exit Neural Networks
- Deep Trajectory Supervision: Deep Supervision Strikes Back
- Latent Reasoning in TRMs is Secretly a Policy Improvement Operator
- Utilizing Multi-Step Loss for Single Image Reflection Removal
- Stacked Deconvolutional Network for Semantic Segmentation