Boundary-Gated Bidirectional Recurrence for Long-Tailed Action Segmentation

Temporal action segmentation means labeling every frame of a long, untrimmed video (say, someone making breakfast) with the action being performed. Models for this task look both backward and forward in time, because whether a frame belongs to "crack egg" or "stir" often becomes clear only from what happens next. The same two-sided context also has a cost. When a rare, two-second action sits between two long, common actions, the model's view of that short interval is mostly made of frames from its frequent neighbors. The rare action then gets "swallowed": its frames are labeled as the neighbor. Recent work on long-tailed segmentation documents this failure and attacks it by reweighting the training loss toward rare classes and rare transitions. We argue the failure is partly architectural. The model lets context flow freely across action boundaries, and reweighting the loss cannot stop a short segment's representation from being dominated by a long neighbor.

We propose a bidirectional recurrent segmenter whose forward and backward memory is attenuated at predicted action boundaries. Each segment, short or long, is then represented mostly by its own frames plus a controlled amount of neighboring context. The boundary detector that drives the gate is class-agnostic. It learns from every transition in the data, including those between rare classes, so it suffers much less from the long tail than the action classifier does. The central claim is testable: on long-tailed segmentation benchmarks, regulating cross-boundary context should recover short tail segments that loss reweighting alone leaves swallowed, and the two remedies should stack.

Core Claim and What Is New

The closest work is Cost-Sensitive Learning for Long-Tailed Temporal Action Segmentation, and it motivates this proposal. It identifies a class-level bias and a transition-level bias: rare transitions get suppressed and brief tail actions are absorbed by frequent neighbors. Its remedy has two parts, both outside the architecture. During training, a Lagrangian min-max loss raises the weight of lagging classes and transitions. At inference, a segment-level nearest-class-mean step smooths predictions. Its backbones (MSTCN, ASFormer, DiffAct) are left unchanged.

One correction to the research file matters here. These backbones are not purely causal. Dilated temporal convolutions and non-causal attention already see future frames. So the real question is not "bidirectional or not." It is "how context crosses boundaries," and this framing makes the proposal sharper. Our contribution is a mechanism that changes how far a frame's memory reaches across a boundary, plus a diagnostic that measures that reach directly. Neither has been tested under class imbalance.

Three bodies of work shape the design:

  • Context helps at transitions. Bidirectional LSTMs catch outbreak reversals that causal models miss (Tzu‐Hsi Song et al.). They also help classify sleep stages that are defined by transitions (García Ciudad et al.). This is why we keep two-sided context rather than abandon it.
  • Boundaries can be detected bottom-up. BSN predicts start, end, and actionness probabilities with small temporal convolutions. We reuse that idea as the gate signal, not for generating proposals.
  • Short and long durations need different treatment. The micro/macro hierarchy in HST-LSTM shows that brief events dilute into background without it: removing the hierarchy dropped UCF101D frame-mAP from 82.4% to 63%. That work was evaluated only on balanced class subsets. We test whether a duration-robust design matters specifically for tail classes.

Mechanism

The model runs on frame features from a frozen spatio-temporal backbone, using the standard pre-extracted features each benchmark provides. Spatio-temporal feature extraction is therefore a fixed input here, not something we redesign. This keeps the comparison with prior segmenters fair.

A small boundary head predicts, for each frame $t$, a probability $b_t \in [0,1]$ that an action boundary lies there. It is trained against ground-truth boundaries with a class-agnostic loss.

A bidirectional linear recurrence then runs in both directions, in the spirit of the bidirectional linear recurrent models line. In the forward direction, each step's carried memory is damped by the boundary probability:

$$h^{f}_t = (1 - \alpha\, b_t)\, A\, h^{f}_{t-1} + B\, x_t ,$$

The symbols are:

  • $x_t$ is the frame feature at time $t$.
  • $h^{f}_t$ is the forward state.
  • $A$ and $B$ are learned matrices.
  • $\alpha \in [0,1]$ is a learned scalar that sets how strongly a boundary cuts memory.

The backward state $h^{b}_t$ is identical except that it runs from the end of the video toward the start.

This gating has a useful property with respect to duration. A two-second segment and a two-minute segment are each summarized mostly by their own frames, so short tail actions are not outweighed simply because they are short. That addresses the dilution problem without building a pyramid of fixed window sizes.

Some actions are only identifiable from the broader activity, for example telling "cooking spaghetti" from "washing dishes" (Benmokhtar). To keep that information, a second, ungated recurrence runs at a coarse temporal stride. Its output is combined with the gated stream through a learned per-frame mixing weight. Frame classification uses the combined representation and is trained with standard cross-entropy, optionally plus the cost-sensitive loss from Pang et al.

Two failure risks are built into the design. First, a missed boundary leaks context; the mixing weight and the soft gate make that leak partial rather than catastrophic. Second, a spurious boundary fragments a segment; this is penalized by the usual over-segmentation metrics.

Research Program

Diagnostic. Before building anything, we quantify the swallowing problem as a property of context flow. On trained baselines, we run a counterfactual neighbor test. We take each tail segment and replace its preceding or following neighbor segment with a different action segment drawn from another video. We then measure how much the tail frames' internal representations and predicted labels change.

If swallowing is driven by context leakage, sensitivity should be highest for short tail segments next to long head segments. It should be low for long segments of any class. We also report a swallowing rate: the fraction of ground-truth tail segments with less than 50% overlap with any predicted segment of the correct class. We stratify it by class frequency and by segment duration.

Method study. We train the boundary-gated model and compare it under matched conditions with reweighting-only remedies, with ungated bidirectional and causal variants, and with combinations of these. Each variant goes through the same diagnostic, so that any accuracy gains can be traced to reduced cross-boundary sensitivity rather than to unrelated capacity.

Baselines

The naive reference is MSTCN trained with plain cross-entropy on the standard features. Any useful method must beat it on tail-bucket metrics without losing more than about one point of overall frame accuracy.

The strongest existing method is MSTCN, and separately ASFormer, trained with the Adaptive Cost-Sensitive Loss plus segment nearest-class-mean inference from Cost-Sensitive Learning for Long-Tailed Temporal Action Segmentation. It is the strongest competitor because it targets exactly the transition-level swallowing we study. The figures the file gives for it are relative only:

  • an 8.1% per-class F1 improvement on Breakfast with an MSTCN backbone;
  • tail-class accuracy described as "nearly doubled" on Assembly101.

No absolute per-class F1 or tail-accuracy value was found on Lacuna for this method on Breakfast, Assembly101, or 50Salads. We will therefore rerun it ourselves with the same features, splits, epoch budget, and hyperparameter search budget as our model, and report the reproduced numbers alongside its reported relative gains. Beating it means higher mean per-class F1 and higher tail-bucket F1@25 on Breakfast and Assembly101 across three seeds, with the gap larger than seed variance.

The key ablations isolate the gate:

  • the same bidirectional recurrence with the gate removed ($\alpha = 0$), with matched parameters;
  • a causal-only version, forward pass only;
  • a gate driven by randomly placed boundaries at the predicted boundary rate;
  • a gate driven by ground-truth boundaries, which gives an oracle upper bound on how much regulating context can help.

Finally, our model combined with the cost-sensitive loss tests whether the two remedies are complementary.

Why not just…?

Why not just apply the cost-sensitive loss to an existing two-sided backbone? This is the real threat, and it is the main comparison. The argument against it as a full solution is mechanical. Reweighting raises the gradient on tail frames, but it does not change the fact that a short segment's representation is computed mostly from neighbor frames. The counterfactual neighbor test measures this directly. If the reweighted baseline already shows low sensitivity to neighbors on short tail segments, and our gate adds nothing on top of it, the architectural claim is wrong and we will say so.

Why not just use a causal model to avoid backward leakage? Prior evidence says future context is what resolves transitions (Song et al., García Ciudad et al.). A causal model also still leaks forward from the preceding head segment. The causal ablation tests both points. We expect lower boundary accuracy and only half the reduction in leakage.

Why not just reweight static frames with class-balanced or focal losses? In VideoLT, class-balanced loss reached 51.1% mAP, below the 51.6% plain baseline. Pang et al. report that focal loss and logit adjustment ignore transition structure. We include class-balanced cross-entropy as a cheap baseline, but do not expect it to compete.

Why not just add multi-scale temporal pyramids or longer dilations? Pyramids address duration (MM-Pyramid, Pyramid Non-Local). But coarse pyramid levels mix neighboring segments even more, which is the opposite of what tail segments need. The coarse ungated stream in our model is effectively that pyramid. Its separate contribution, and the effect of removing the gated stream, show which part does the work.

Proposed Experiments

Data. We use Breakfast, 50Salads, and Assembly101 with their standard features and splits, partitioning classes into head, medium, and tail buckets by frame count.

Metrics. We report:

  • frame accuracy;
  • edit score;
  • F1 at overlap thresholds 10, 25, and 50;
  • mean per-class F1;
  • tail-bucket F1;
  • the swallowing rate, stratified by segment duration.

Decisive check. Three conditions must hold together:

  1. On short tail segments, the gated model shows clearly lower counterfactual neighbor sensitivity than both the ungated bidirectional model and the cost-sensitive baseline.
  2. That reduction comes with a lower swallowing rate and higher tail F1.
  3. Combining the gate with the cost-sensitive loss beats either one alone.

Secondary check. The oracle-boundary gate should show a large gain over the ungated model. If even the oracle gate barely helps, context leakage is not the bottleneck and the proposal fails cleanly. If the oracle helps but predicted boundaries do not, the remaining work lies in detecting boundaries for rare transitions. In that case we will measure boundary-detection recall separately for transitions involving tail classes.