Fuzzy Containment Between Steps and Phases in Surgical and Assembly Video

Long procedural videos, such as a two-hour gastric bypass or an hour of toy assembly, are labeled at two levels: coarse phases, and fine steps that happen inside them. Current models predict both levels from a shared network and hope the outputs agree. They often don't. A step gets labeled in a phase where it cannot occur, or a phase boundary lands in the middle of a step. We propose to represent every predicted segment, at both levels, as a fuzzy interval: an earliest and latest plausible start plus an earliest and latest plausible end, with plausibility fading linearly between them. A small differentiable layer then passes bound-tightening messages along a graph whose edges say "this step lies inside that phase." Phase bounds clip step bounds, and confident step bounds pull phase bounds. The result is decoded back to frame labels and trained end to end.

The question is not whether constraints help in general; earlier work shows they do. We ask which kind of temporal knowledge survives when a model moves to a new hospital. Across institutions, surgical teams reorder and skip steps (their workflow "dialects"), so rules about step order that are mined at one center can mislead a model at another. The fact that a given step belongs to a given phase is part of the annotation taxonomy, so it should hold everywhere. A model trained on one Swiss center and tested on a French one drops to roughly 33% phase F1 (Challenges in multi-centric generalization). That drop gives a clean place to test the claim that fuzzy, two-way containment transfers across centers and crisp ordering does not.

Core Claim

Our claim has three parts. First, building fuzzy interval bounds so they are valid by construction, and propagating them across a two-level step/phase graph, reduces boundary jitter and cross-level contradictions compared with sharing a backbone between the two levels. Second, the benefit comes from messages flowing in both directions, not only from phases down to steps. A hard top-down mask, which forbids step labels incompatible with the current phase, is the obvious cheap alternative, and it should lose because it passes every phase error on to the steps. Third, under cross-center shift, containment constraints should keep their value while order constraints mined at the training center should lose value or even hurt.

How This Relates to Prior Work

The interval representation comes from classical temporal reasoning. Rit's Sets of Possible Occurrences places an uncertain event in a start-by-end plane and shrinks that feasible region by repeated arc-consistency filtering. We unroll a fixed number of such tightening sweeps as soft-min and soft-max messages, which makes them differentiable. Allen's interval algebra introduced reference intervals, enclosing intervals that keep constraint propagation local. Our phases play exactly that role for their steps, so propagation cost scales with the number of steps per phase rather than with all pairs of segments. Godo and Vila showed that the Łukasiewicz implication, $\min(1, 1-a+b)$, gives a continuous degree of entailment without sudden collapses. We use it to score how far a step is contained in a phase.

The closest neural precedent is Wen et al.. They propagate fuzzy start/end 4-tuples over a flat, day-level event graph in text with graph attention. Their predictions are unconstrained, and about 4% violate interval geometry, for example a latest start that falls before the earliest start. We differ in three ways: our parametrization rules out such violations, our graph has two levels rather than one, and we work at frame resolution in video.

Three lines of work motivate the gap. MTMS-TCN predicts 12 phases and 46 steps through a shared temporal convolutional network, with no containment logic. Differentiable Temporal Logic (DTL) adds smooth losses for crisp ordering rules to segmentation models and gains 3.6 Edit points on 50Salads over ASFormer. DTL also constrains our design, because its authors report that piling on many mined rules adds gradient noise. That finding is one reason we keep a small, structural rule set instead of thousands of ordering rules. Task-graph learning (TGML) learns crisp precondition graphs and admits that optional steps cause false alarms.

GTRM reasons over segment nodes, but it builds them from hard frame predictions, so early mistakes carry forward. Our segment nodes keep soft bounds throughout. None of these works couples two annotation levels through graded interval relations.

Research Program

A temporal backbone, either ASFormer or MS-TCN, produces frame features. Two heads propose segments, one per level, as frame-wise class scores together with soft start and end distributions. From these we take the top K overlapping hypotheses per level. Each hypothesis $i$ carries a 4-tuple $(s^-_i, s^+_i, e^-_i, e^+_i)$, the earliest start, latest start, earliest end and latest end. We parametrize it as a start point followed by cumulative softplus increments, so the four values are always in order and geometric violations cannot occur.

A frame at time $t$ belongs to segment $i$ with trapezoidal membership:

$$\mu_i(t)=\min\Big(1,\ \tfrac{[t-s^-_i]_+}{s^+_i-s^-_i+\epsilon},\ \tfrac{[e^+_i-t]_+}{e^+_i-e^-_i+\epsilon}\Big)$$

Here $[x]_+=\max(0,x)$ and $\epsilon$ is a small constant.

Containment edges link each step hypothesis $j$ to each overlapping phase hypothesis $p$ that the taxonomy allows. The graded degree to which $j$ lies inside $p$ is:

$$c_{jp}=\min\Big(1,\ 1-\tfrac{[s^-_p-s^-_j]_+ + [e^+_j-e^+_p]_+}{\tau}\Big)$$

The tolerance $\tau$ is learned. As $\tau\to 0$, this reduces to crisp Allen "during."

Each message-passing sweep does two things. It moves step bounds toward the bounds of the phase they are contained in, and it moves phase bounds toward a confidence-weighted soft-min of the starts and soft-max of the ends of the steps they contain. After three to five sweeps, we decode frame labels as the class scores weighted by membership. The loss combines frame cross-entropy at both levels with a penalty on $1-c_{jp}$ for the ground-truth pairing. An optional third edge type carries fuzzy metric delays between steps within a phase, of the form "B starts roughly 30 to 120 seconds after A." These delays are fitted as trapezoids from training durations, in the spirit of the bimanual fuzzy-Allen work, which fit relations offline. We learn ours inside the network.

The spatio-temporal graph enters in a limited form, and only where the data supports it. On Assembly101, step hypotheses are scored by a per-frame hand-object graph built from the dataset's 3D hand poses and object tracks, in the style of Stacked-STGCN. The motivation is the Assembly101 finding that hand poses alone reach 28.7% recognition accuracy while poses fused with object labels reach 62.0%. The surgical experiments use frame features only, because MultiBypass140 provides no tool-anatomy tracks.

Baselines

The naive reference point is MTMS-TCN as published: a ResNet-50 feature extractor, a multi-stage temporal convolutional network, and joint prediction of phases and steps. Its reported numbers are about 90% phase accuracy when trained and tested at the same center, about 33% phase F1 when trained on Bern and tested on Strasbourg, and 81.9% phase F1 on Strasbourg when trained on both centers (Challenges in multi-centric generalization). We found no reported step-level result on Lacuna for MultiBypass140, so we will rerun the step numbers ourselves. We retrain MTMS-TCN on identical features and splits so that every method shares the same backbone.

The strongest single-level competitor is ASFormer trained with DTL losses. ASFormer reports 76.0 F1@10, 75.0 Edit and 73.5% accuracy on Breakfast, and DTL adds 3.6 Edit points to ASFormer on 50Salads. We found no reported DTL number on Lacuna for Breakfast. To make this baseline as strong as possible, we give DTL step-in-phase containment rules written as temporal-logic formulas in addition to its mined ordering rules. That way the comparison tests our representation, not merely access to the taxonomy.

The key ablation is the hard top-down mask: the same backbone, with step logits zeroed wherever the predicted phase forbids them. Our layer approaches this mask as $\tau\to 0$ with downward-only messages. We will also run top-down-only fuzzy messages, and an unconstrained version that regresses 4-tuples with a free head, as Wen et al. do, to measure how often violations occur in video.

All methods use the same features, the same splits, and the same budget of 20 hyperparameter trials. They get matched parameter counts within 5%, with extra backbone channels added where needed. We count our method as winning if it beats the strongest baseline by at least 2 points in Edit and F1@50 at both levels, averaged over three seeds, with gains larger than the variation across seeds. The cross-center test is the deciding one, measured as Bern-to-Strasbourg step and phase F1.

Why not just…?

Why not just mask step predictions by the predicted phase? This alternative is the most serious threat. Within a single center, where phase accuracy is about 90%, the mask may come within a point of our method. Our argument is that under transfer the phase predictions themselves are poor, about 33% F1, and the mask then forces step labels to follow wrong phases. In our method, confident step evidence can move phase boundaries instead. The comparison of masking, top-down-only messages, and two-way messages under cross-center testing settles this. If two-way messages show no advantage there, the fuzzy layer has failed to justify itself, and we will report that.

Why not just add containment rules to DTL? DTL encodes "during" as qualitative orderings that are either satisfied or violated. It has no notion of a boundary that is roughly right, and its authors observed gradient noise as rules accumulate. We include DTL with containment rules as a baseline. If it matches our method, then the value lies in the rules rather than the interval representation, and that would be a useful negative result.

Why not just use a stronger hierarchical backbone such as ASFormer with two heads? ASFormer's window hierarchy enlarges the span of frames each prediction can see, but its two levels never constrain each other. DTL's gain on top of ASFormer suggests that structural errors remain. We measure cross-level violation rate directly, as the fraction of predicted steps that lie outside any allowed phase, alongside segmentation metrics.

Why not run interval propagation as post-processing? Post-hoc filtering cannot send corrections back to the feature extractor, and Wen et al. report that relational GAT propagation outperformed their ILP-based alternative by 7.0% in match rate. We test this directly by freezing the backbone and applying our layer only at inference time.

Proposed Experiments

MultiBypass140 is the primary benchmark. We evaluate within each center, across centers in both directions, and with joint training. Metrics are F1@{10,25,50}, Edit, frame accuracy, boundary error in seconds, and cross-level violation rate, all reported at both levels.

The decisive experiment crosses constraint type with training regime. We train on Bern with containment edges only, with containment plus fuzzy step-delay edges, and with delay edges only, then test on Strasbourg. We predict that containment-only transfers best, and that delay edges help within a center but hurt across centers unless their tolerance widens. If containment shows no transfer advantage, the main claim is refuted.

Assembly101 coarse and fine segments provide a second, much larger nested hierarchy. There we run the hand-object graph variant against a frame-only variant. We found no reported Lacuna result for Assembly101 two-level segmentation, so all of its baselines will be rerun. Breakfast and 50Salads are used only to check that the fuzzy within-level delay edges do not fall below the published ASFormer and DTL numbers on flat benchmarks.

Finally, we sweep the learned $\tau$ toward zero to confirm that gains vanish as containment becomes crisp. That outcome would tie the improvement to graded boundaries and not to the added graph capacity.