Learning Step Order from Text, Deploying It on Video

Instructional videos are expensive to annotate step by step, but the order of procedural steps is written down for free all over the place: in recipes, how-to articles, and the step lists that accompany the videos themselves. This proposal asks whether a video model can learn which step comes before which mostly from text, and then use that knowledge on video of tasks it has never seen ordered on screen. The obstacle is the cross-modal representation gap. Even in a shared vision-language embedding space, text and video features sit in separate regions, so an ordering rule learned on text inputs is not automatically valid for video inputs. We propose to close that gap only where ordering decisions are made, using an adversary that is present during training and removed before deployment, so that the shipped video model costs no more to run than a plain one.

Concretely, during training a pairwise "does step $i$ precede step $j$" head reads relation features computed from pairs of step embeddings, drawn from both text steps and video clips. A domain discriminator tries to tell whether each relation feature came from text or from video. A gradient reversal layer, which flips the sign of the discriminator's gradient before it reaches the encoder, pushes the encoder to make text pairs and video pairs indistinguishable at the level the ordering head sees. At test time the text branch, the pairwise head, and the discriminator are all discarded. A single-pass video model assigns each clip a scalar position score, and the clips are sorted by that score. The bet is that this transfers ordering knowledge to video tasks that appeared only in text during training, and that it does so better than closing the gap globally or chaining a captioner to a text orderer.

Core Claim

Two claims are tested. The first is that the part of the modality gap that hurts transfer of order knowledge is located in pairwise relation features rather than in individual step embeddings. The second is that adversarially aligning those relation features across modalities in a training-only scaffold lets a cheap video orderer recover step order on held-out procedural tasks that it saw only as text. If the claims hold, aligning relation features should beat aligning per-step embeddings, and both should beat a shared contrastive space used as is. The deployed model should also lose little when the pairwise head is removed. If simple global gap-closing (mean shift or a distribution-matching penalty) does as well, the adversarial component is unnecessary, and we will report that.

Relation to Prior Work

Three procedure-planning systems motivate the setting and define the incumbents. P3IV predicts action sequences between start and goal frames. It works in a pretrained vision-language space and uses a non-autoregressive decoder. Its adversary, however, is a critic that makes generated plans look human-like; it does not make the inputs domain-invariant. E3P infers an event label to build text prompts and masks action relations to avoid memorized sequences. Its authors name cross-event generalization as unsolved, and that gap is our target. VideoPlan is the closest in spirit. It adds training-only auxiliary tasks and multi-token heads that are dropped at test time. It does not perform adversarial alignment and does not use text-only tasks as a source of order supervision. PDPP and Dual Dynamics Networks depend on iterative denoising or beam search at test time. That runtime cost is exactly what we want to avoid.

The ordering machinery comes from text sentence ordering, and we reuse its techniques. Topological Sort for Sentence Ordering predicts all pairwise precedences and then sorts them. Enhancing Pointer Network for Sentence Ordering with Pairwise Ordering Predictions and BERSON show that pairwise relational modules fix the error buildup of pointer decoders. All three keep those modules at inference. We keep them only as training signal. Video Clip Order Prediction shows that order prediction over clips teaches useful video features, but it classifies among $N!$ permutations within single-domain video. We instead score pairs and transfer across modalities.

The zero-cost design follows training-only scaffolds. BatchFormer trains a shared classifier on features both with and without an auxiliary module so the module can be unplugged. We copy this dual-path trick to limit the representation shift when the pairwise head is removed. REPA supplies a caution: alignment losses are sensitive to the depth at which they are attached, so attachment depth will be swept.

The gap-closing literature both motivates and constrains the design. ReViSE argues that instance-level contrastive matching leaves the two modality distributions apart and closes the gap with a distribution-matching penalty (maximum mean discrepancy). CaReFlow maps one modality's distribution onto another with rectified flow and finds that aggressive one-to-one alignment erases modality-specific detail. Both align whole embeddings. We align only relation features, partly to avoid the over-alignment that adversarial adaptation is known to cause in unimodal settings, as in FAN-Net and Ren et al., where structural constraints were needed to stop discriminative information from collapsing.

Research Program

The deployed model is a small permutation-equivariant set encoder over clip features from a frozen video backbone (MIL-NCE/HowTo100M features, as used by P3IV and PDPP), followed by a linear position scorer. It takes clips in shuffled order and produces step embeddings $h_i$ and scores $s_i$ in a single pass, and sorting by $s_i$ costs only $O(N \log N)$. During training, the same encoder also embeds text step descriptions through the frozen text tower of the same vision-language model, plus a light adapter. A relation network forms $r_{ij} = g(h_i, h_j)$ for every pair, and a pairwise head predicts precedence from $r_{ij}$. A discriminator $D$ reads $r_{ij}$ through a gradient reversal layer. The training objective is

$$\mathcal{L} = \mathcal{L}_{\text{list}}(s; \text{video}) + \alpha\, \mathcal{L}_{\text{pair}}(r; \text{text} \cup \text{video}) - \lambda\, \mathcal{L}_{D}(r),$$

where $\mathcal{L}_{\text{list}}$ is a listwise ranking loss on the per-clip scores, $\mathcal{L}_{\text{pair}}$ is binary cross-entropy on precedence, $\mathcal{L}_D$ is the discriminator's text-versus-video cross-entropy, and $\alpha, \lambda$ are weights. The minus sign on the last term is implemented by the gradient reversal layer, and $\lambda$ is ramped up over training to avoid early instability. The listwise loss also receives a text-path term through the shared scorer, which is the BatchFormer-style dual path, so the scorer never depends on the pairwise head.

The split that makes the question sharp is a split over tasks. In CrossTask and COIN we hold out a subset of tasks whose videos are never seen with order labels. Their step sequences appear only as text, taken from the datasets' own step annotations. Optionally we add sentence-ordering text from ROCStory and scientific abstracts as generic discourse pretraining. At evaluation, the model orders shuffled ground-truth step clips from held-out-task videos, and separately performs standard start/goal procedure planning. A second shift tests third-person to egocentric transfer: train on CrossTask and COIN, then test on CaptainCook4D clips using the task-graph codebase from Differentiable Task Graph Learning. That paper's handling of optional and repeated steps also tells us which evaluation sequences to treat as having multiple valid orders.

The analysis has three parts. We measure text-versus-video separability of $h_i$ and $r_{ij}$ with a probe, before and after training. We sweep the depth at which $r_{ij}$ is formed. We compute the difference in accuracy with the pairwise head kept versus removed, which measures the representation shift caused by detaching the scaffold.

Baselines

The naive reference point is the same deployed video orderer trained only on seen-task video with the listwise loss, with no text and no pairwise head. Comparing against it shows whether text adds anything for held-out tasks. A second reference, the "free gap" baseline, trains the pairwise head on text in the frozen MIL-NCE space and applies it directly to video pairs. This tests whether the pretrained shared space already suffices. Both use identical features, encoder size, and training steps.

The strongest existing methods on the planning metric are E3P and VideoPlan. The highest absolute figure in the file is from E3P: 26.40% success rate at horizon $T=3$ on CrossTask, and 19.57% at $T=3$ on COIN. VideoPlan and PDPP report only relative gains in the file; no absolute success rate for them is found on Lacuna. We will rerun E3P and P3IV from their released settings on our task-held-out split. That split is new, so no reported result exists for it, and the published 26.40% applies only to the standard split. On the standard split, beating the method means exceeding 26.40% success rate at $T=3$ on CrossTask with a single-pass planner head built on our encoder. On the held-out split, beating it means a higher success rate and a higher Kendall's $\tau$ than E3P under the same features and tuning budget. For clip ordering metrics (Kendall's $\tau$ and perfect match ratio on video), no reported result is found on Lacuna for any method, so all numbers will come from our own runs.

Ablations isolate the key ingredient. We compare the full model against four variants. The first removes the discriminator, so text and video train the pairwise head jointly without an adversary. The second places the discriminator on per-step embeddings $h_i$ rather than on relation features. The third replaces the adversary with a maximum mean discrepancy penalty on $r_{ij}$, in the spirit of ReViSE. The fourth replaces it with mean-centering of each modality. All variants get equal hyperparameter search budgets, the same number of trials over $\alpha$, $\lambda$, and attachment depth. An upper-bound variant keeps the pairwise head and topological sorting at test time. This shows what the zero-cost constraint gives up, and we report its latency alongside its accuracy.

Why not just…?

Why not just use the pretrained vision-language space, which is already aligned? P3IV and E3P rely on exactly this, and ReViSE and CaReFlow both report that contrastively trained spaces leave the modalities in separate clusters. Still, whether the residual gap matters for ordering is an empirical question. The "free gap" baseline answers it directly. If a text-trained pairwise head already orders video pairs well in the raw MIL-NCE space, the proposal's premise fails.

Why not just close the gap globally with mean-centering or a distribution-matching penalty? This is the most serious threat, because those methods are simpler and more stable than a minimax game. Our reason to expect a difference is that order depends on how two steps relate, not on where each one sits. A global shift can align marginals while leaving relation features mismatched. The ablation that applies maximum mean discrepancy to $r_{ij}$ already separates "align relations" from "use an adversary". If it matches the adversarial version, we will adopt it and frame the contribution around relation-level alignment.

Why not just recognize each clip's step label and order the labels with a text sentence orderer or language model? This two-stage pipeline needs no alignment at all. It fails in two places. Step recognition errors propagate, and the held-out tasks have no video-labeled step classes to recognize. It also adds a text model at inference, which violates the cost constraint. We will run it anyway, with an oracle-label version as its ceiling. If it beats us with oracle labels but not with predicted labels, that locates the bottleneck in recognition rather than in ordering.

Why not just add VideoPlan-style auxiliary heads on video alone? That approach improves supervision on seen tasks but provides no information about tasks absent from video. On the held-out split it should match the naive baseline. If instead it improves held-out ordering, generic auxiliary supervision rather than text transfer explains the gains, and our ablation without the discriminator will show this.

Decisive Checks

The central experiment is held-out-task clip ordering on CrossTask and COIN, measured by Kendall's $\tau$ and perfect match ratio. Text-relation adversarial alignment must beat the naive baseline, the "free gap" baseline, and the per-step-embedding adversary. The probe should show that relation features become modality-inseparable while the per-clip position score stays accurate. Removing the pairwise head should cost less than a small fraction of its gain over the naive baseline, which verifies that the scaffold can be discarded.

Secondary checks cover three things. On the standard split, success rate at $T=3$ is compared against E3P's reported 26.40% on CrossTask and 19.57% on COIN. On the egocentric split, ordering accuracy is measured on CaptainCook4D, where we test whether adding egocentric unlabeled clips as a third domain for the discriminator helps. Finally, we report latency per video against PDPP's iterative denoising and P3IV's reported 6.75 ms. A negative result on the central experiment, with gains appearing only on seen tasks, would mean that text order knowledge does not cross the modality gap through this route, and that would be worth reporting too.