Long videos of people doing tasks, such as making a salad or preparing breakfast, have two clocks. Frames change many times a second, but the actions that matter ("cut tomato", "pour milk") change only every few seconds or minutes. Most models for labeling every frame with its action run a single clock. Some are convolutional stacks over frames; some are attention over windows; some are recurrent or state-space models, which carry a running memory forward one step at a time. All of them update their memory, or their receptive field, uniformly in frames. As a result, a 10,000-frame video forces the model to carry task context across 10,000 updates, even though the task itself has only a dozen or so steps.
We propose a two-level hybrid that gives the slow level its own clock. A bank of short convolutions reads the frames and estimates, at every frame, how likely an action boundary is. That estimate then sets the step size of a slower state-space layer. A state-space model is a linear recurrence whose memory decays at a rate controlled by a step-size parameter. Mamba is the variant where that step size depends on the input. The slow layer's memory is held fixed within an action and rewritten only at boundaries, while a fast layer tracks frame-level detail. The bet is simple: if the coarse memory ticks once per event rather than once per frame, it should forget less over long videos and flicker less inside actions. Both are known failure modes of flat state-space models and convolutional networks on this task.
Core Claim
Hierarchical temporal models so far build their coarse levels with fixed strides: pooling in encoder-decoder convolutional networks, progressive downsampling in multi-scale state-space decoders, or concatenating fixed pyramid levels. The file names the cost of this directly: rigid grids split or blur actions whose boundaries are irregular. Our claim is that the right coarse "downsampling" for dense segmentation can be built inside the state-space recurrence itself. We do this by tying the coarse layer's step size to a convolutionally predicted boundary probability. This produces a coarse level whose effective length is the number of events rather than the number of frames, in a single feedforward pass with no discrete pooling, clustering, or post-hoc parsing.
Mechanism
A discretized selective state-space layer updates its hidden state as
$$h_t = \exp(\Delta_t A)\, h_{t-1} + \Delta_t B_t x_t,$$
where $h_t$ is the hidden state at frame $t$, $x_t$ is the input, $A$ is a learned (stable, negative) transition matrix, $B_t$ is an input projection, and $\Delta_t \ge 0$ is the step size. When $\Delta_t \to 0$, the update becomes the identity: the state is held exactly and receives no input. In the coarse layer we set
$$\Delta^{c}_t = \Delta_{\max}\, b_t, \qquad x^{c}_t = s_t,$$
where $b_t \in [0,1]$ is the boundary probability from the convolutional front-end, $\Delta_{\max}$ is a learned scale, and $s_t$ is the fast layer's running summary of the current segment. The coarse state therefore absorbs a whole segment's summary at the moment that segment ends, and otherwise sits still.
Front-end. The convolutional front-end reuses the parallel multi-kernel design of MHSNet. That paper runs depthwise convolutions with kernels 3, 5, 7 and 9 upstream of a Mamba block for machinery fault diagnosis. Here, those kernels operate over pre-extracted frame features and emit both local features and the boundary logit.
Fast layer and readout. The fast layer is an ordinary bidirectional selective state-space layer at frame rate. For directional context we use the decomposed bidirectional block from Video Mamba Suite, which shares one state-space core between forward and backward scans. Both coarse scans are run, forward and backward, and their states are broadcast back to every frame through a skip connection. Frame classification therefore reads fine detail and event-level context together.
Supervision. Boundaries are supervised with transitions in the ground-truth labels, which segmentation datasets provide for free. They also receive gradient end-to-end through $\Delta^c$. We add the truncated smoothing loss of MS-TCN.
Why decay should shrink. The quantitative expectation follows from the update rule. The coarse memory's decay between frames $t_1$ and $t_2$ is $\exp\big(A \sum_{t_1}^{t_2} \Delta^c_t\big)$. Its exponent scales with the number of predicted boundaries in between, not with $t_2 - t_1$. Context from early in a Breakfast video should therefore survive in proportion to the number of actions elapsed, rather than the number of frames.
Relation to Prior Work
Work that motivates the proposal by exposing the gap.
- MambaTAD identifies "temporal decay" (forgetting early context) and "self-element conflict" (self-similarity washing out boundaries) as the two failure modes of state-space models on untrimmed video. It fixes them with diagonal masking and a head that concatenates fixed pyramid levels into one flat sequence. We attack decay by changing the clock rather than the mask, and we target per-frame segmentation rather than detection.
- Activity Grammars for Temporal Action Segmentation shows that both MS-TCN and ASFormer make "out-of-context" errors that need an external grammar parser. It also concedes that the parser cannot recover from bad base predictions. We test whether an event-level memory reduces those errors inside the network.
Work that constrains the design.
- The encoder-decoder temporal convolutional network of Temporal Convolutional Networks for Action Segmentation and Detection beat a Bi-LSTM on 50 Salads (64.7% vs 55.7% accuracy) while training in about a minute versus thirty. Any recurrent component must therefore be parallelizable. A selective scan is, and a classic LSTM is not.
- ASFormer builds a hierarchy by doubling window size and dilation per layer, but stops improving beyond nine blocks. This is a ceiling a recurrence-based hierarchy should not share.
- ViS4mer shows that multi-scale structured state-space decoders cut memory 8× against attention. However, it downsamples on a fixed schedule and only predicts clip-level labels.
Work that supplies technique. MHSNet supplies the front-end, Video Mamba Suite supplies the bidirectional block, and MS-TCN supplies the smoothing loss and multi-stage refinement. What none of these do is let a predicted event signal set the update rate of a recurrence.
Research Program
Phase one: architecture and analysis. We build the two-level model, plus an optional third level clocked by a second, sparser boundary head intended to track activity-level phases. We then run analyses that test the mechanism rather than just the score:
- We measure how much the coarse state at a late frame depends on features from early frames, via gradient norms and feature-ablation probes. We plot this against distance in frames and in events, for our model and for a flat bidirectional Mamba. If the mechanism works, our curve should be flat in frames and decay only in events.
- We measure the distribution of $\sum \Delta^c$ within ground-truth segments, to verify that the clock actually freezes inside actions.
Phase two: robustness of the clock. We study what happens when the clock is wrong. We inject boundary noise at test time (missed, spurious, and displaced boundaries) and measure degradation. We compare a hard clock (thresholded $b_t$) against the soft one, and teacher-forced ground-truth boundaries against predicted ones. Teacher-forced boundaries give an upper bound that tells us how much of any gap is boundary-prediction error rather than architecture.
Baselines
Naive reference. The do-nothing reference is the per-frame linear classifier on the same I3D features, plus the encoder-decoder temporal convolutional network. For the latter, the cited paper reports 64.7% frame accuracy on 50 Salads. These anchor how much any temporal modeling helps.
Strongest convolutional and attention competitors. We include MS-TCN and ASFormer, the two networks the grammar paper also uses. ASFormer is the strongest single-model competitor with numbers in the file. It reports F1@10 85.1%, Edit 79.6% and accuracy 85.6% on 50 Salads; F1@10 90.1%, Edit 84.6% and accuracy 79.7% on GTEA; and F1@10 76.0%, Edit 75.0% and accuracy 73.5% on Breakfast. The grammar paper reports a lower ASFormer Edit of 76.5 on 50 Salads in its own runs, rising to 79.9 with KARI parsing. That gap is why we will rerun ASFormer ourselves under our splits and seeds rather than trust a single number. For MS-TCN, no reported result found on Lacuna for exact 50 Salads, GTEA or Breakfast scores. ActFusion reports an average F1 of 88.5% on 50 Salads. It is a diffusion decoder with iterative sampling, so we report it as a reference ceiling rather than a like-for-like competitor.
Fairness protocol. All baselines use the same I3D features, the standard cross-validation splits, matched parameter counts within 10%, the same smoothing loss where applicable, and the same hyperparameter search budget of a fixed number of trials per model.
Ablations that isolate the key ingredient. The first ablation is a flat bidirectional Mamba with the same front-end and refinement. No reported result found on Lacuna for a flat Mamba on 50 Salads, GTEA or Breakfast segmentation, so we train it. The second, and decisive, ablation is a fixed-stride hierarchy: identical layers, but with the coarse state-space layer run on features pooled at a fixed stride of 8 or 16 instead of event-clocked. This is the "obvious" multi-rate pyramid of the file's first gap. Beating it is the core claim; beating ASFormer is secondary.
What counts as beating a baseline. We require higher Edit and F1@50 than both ASFormer (rerun) and the fixed-stride hierarchy on Breakfast, beyond seed variance across five seeds. Edit and F1@50 penalize flicker and boundary misplacement, which are exactly the errors the clock targets. We additionally require a lower out-of-context transition rate, measured against transitions observed in training videos.
Why not just…?
Why not just use a fixed-stride multi-scale state-space pyramid? This is the most direct threat. It is the natural extension of ViS4mer to dense prediction, and it inherits parallelism and linear cost. Our argument is that fixed strides split irregular actions, but that is a hypothesis, not evidence. The fixed-stride ablation above settles it. If it matches the event-clocked model on Breakfast Edit and on the decay-versus-events curve, the clock adds nothing, and we will report that.
Why not just predict boundaries, pool features within segments, and run a coarse model on the segment sequence? This hard two-stage version is close to our model with a thresholded clock, and it may work as well. We include it as a baseline: the same front-end, hard segment pooling, and a coarse bidirectional Mamba over segments, broadcast back. Our expected advantage is that soft clocking lets boundary uncertainty be corrected by downstream gradient. The boundary-noise experiment tests exactly this: a hard pipeline should degrade sharply with missed boundaries, while the soft clock should degrade gracefully. If it does not, the simpler pipeline wins and the contribution reduces to the analysis.
Why not just keep ASFormer or MS-TCN and add grammar post-processing? The grammar paper shows parsing helps Edit, from 76.5 to 79.9 for ASFormer on 50 Salads. It also states that the parser is bounded by base predictions and adds discrete search. We will apply KARI-style parsing on top of both ASFormer and our model. If our model alone matches ASFormer plus parsing, the memory is doing the parser's job in one pass. If parsing still adds as much to ours as to ASFormer, the two are complementary and our out-of-context claim fails.
Why not just widen the convolutional receptive field? Dilated stacks already reach thousands of frames at linear cost. However, MambaTAD reports an 11.2% mAP gain over a convolution-based detector on long-duration actions, suggesting that bounded effective receptive fields hurt on long instances. We test this directly by stratifying Breakfast results by video length and action duration.
Proposed Experiments and Decisive Checks
Datasets and metrics. Experiments run on 50 Salads, GTEA and Breakfast with pre-extracted I3D features, reporting frame accuracy, Edit, and F1 at overlaps of 10, 25 and 50. Breakfast is the primary test because its videos are longest and most procedural. GTEA, being short, serves as a check that the clock does not hurt when there is little to forget. We also report training time and peak memory against ASFormer, with sequence length scaled by concatenating Breakfast videos, to confirm linear scaling.
Decisive checks. Four results would decide the proposal:
- On Breakfast, the event-clocked model beats the fixed-stride hierarchy on Edit and F1@50.
- Its dependence-on-early-context curve is flat in frames but decays in events, while the flat Mamba's decays in frames.
- Its gain over the flat model grows with video length.
- Under 20% injected missed boundaries, it degrades less than the hard segment-pooling pipeline.
If the first check fails, the idea is wrong. If the first passes but the second fails, the gain comes from something other than the claimed mechanism, and we will say so.
References
- MHSNet: A Multi-Scale Hidden State Interaction Network for Fault Diagnosis of Rotating Machinery
- Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understanding
- MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation
- MambaTAD: When State-Space Models Meet Long-Range Temporal Action Detection
- Activity Grammars for Temporal Action Segmentation
- Temporal Convolutional Networks for Action Segmentation and Detection
- ASFormer: Transformer for Action Segmentation
- Long Movie Clip Classification with State-Space Video Models