Policies that output several future actions at once in a single pass, rather than one action at a time, are cheaper to run. They are also less exposed to errors that compound over a long rollout. They fail in a specific way, though. When there are two equally good ways to act, say going around an obstacle on the left or on the right, each predicted step picks its own option independently. The result is an incoherent chunk that starts left and finishes right. Machine translation hit exactly this failure when it moved to parallel decoding, and fixed it by first predicting a small plan that commits to one mode. This note proposes that a short natural-language subgoal ("go to the yellow box", "open the purple door") can play that plan's role for parallel action decoding. It further proposes that this role explains when language supervision helps a policy at all.
The literature on the value of language in policy learning reports a consistent but unexplained pattern. Language adds nothing on easy tasks, helps most on moderately hard long-horizon tasks, and stops helping when the agent lacks the underlying motor skills. Our claim is narrower and testable. For a parallel decoder, the benefit of conditioning on a subgoal should track one measurable quantity: how much the subgoal reduces the statistical dependence among the actions in a chunk. The benefit should therefore be larger for parallel policies than for step-by-step ones, and it should grow with chunk length. If this holds, the "sweet spot" of instructional utility becomes a measurable property of the demonstration data instead of an after-the-fact observation.
Core Claim
A few terms first. An autoregressive (AR) decoder produces each action conditioned on the ones before it. A non-autoregressive (NAR) decoder produces all positions of a chunk in parallel, each conditioned only on the shared context. Take a chunk of $H$ future actions $A_{1:H}$ and a context $c$ (observation history plus goal). The best any fully factorized parallel decoder can do is match each position's marginal distribution. Its unavoidable excess loss over the true joint distribution is then exactly the total correlation of the chunk:
$$ \mathrm{TC}(A_{1:H}\mid c)=\sum_{h=1}^{H} H(A_h\mid c)-H(A_{1:H}\mid c), $$
Here $H(\cdot)$ is entropy. This is a standard identity: the KL divergence from a joint to the product of its marginals equals total correlation. Adding a subgoal $\ell$ to the context changes this floor to $\mathrm{TC}(A_{1:H}\mid c,\ell)$. We predict that the success-rate gain from language in a parallel policy is monotone in the reduction
$$ \Delta_{\mathrm{TC}}=\mathrm{TC}(A_{1:H}\mid c)-\mathrm{TC}(A_{1:H}\mid c,\ell). $$
We also predict that this relationship is much weaker for AR policies, which can coordinate positions on their own. Both total correlations can be estimated on held-out demonstrations. We fit an AR chunk model, which gives a tractable joint likelihood, and per-position marginal heads, and take the difference of their log-likelihoods. Under this account, the "redundancy regime" is where $\Delta_{\mathrm{TC}}\approx 0$. That covers two cases: the chunk has only one sensible mode, or the subgoal is already implied by the goal.
Relation to Prior Work
Three papers motivate the proposal by documenting the utility pattern without explaining it. Improving Long-Horizon Imitation through Instruction Prediction adds an autoregressive instruction-prediction head to a behavior-cloning policy and discards it at test time. It finds large gains on BabyAI BossLevel and negligible gains on ALFRED, where instructions follow from the goal. Think Before You Act interleaves subgoal captions with actions in one GPT-2-style autoregressive model, and finds the gap over a caption-free model widening from PutNextLocal to BossLevel. Teaching Embodied Reinforcement Learning Agents finds language feedback helps most at moderate difficulty. All three use autoregressive, one-action-at-a-time decoders. None asks whether the decoding factorization changes how much language is worth, and that question is our contribution.
The method comes from parallel sequence generation. Non-Autoregressive Neural Machine Translation introduced a latent "fertility" plan to commit parallel decoders to one output mode, along with sequence-level distillation from an AR teacher. Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement supplies the few-pass refinement decoder we use as an intermediate between one-shot and fully sequential decoding. Thinking Clearly, Talking Fast is the closest structural analogue: it plans concept keywords, then fills text in parallel. It works on dialogue with a static concept graph, however, and never measures when the plan is redundant. We replace the concept graph with environment subgoals and make redundancy the object of study.
Two papers constrain the design. Promises and Pitfalls of Generative Masked Language Modeling and Why Diffusion Language Models Struggle with Truly Parallel Decoding? analyze why dependence between positions breaks parallel decoding, which is the lens behind our total-correlation predictor. Bringing Stability to Diffusion reports much higher training-gradient variance for masked diffusion than for AR models. That is why we treat masked discrete diffusion over action chunks as one variant rather than the default.
Two lines of work on actions come close without language. Goal-Conditioned Imitation Learning using Score-based Diffusion Policies generates whole action sequences in parallel, and Chain-of-Action and CDP argue for autoregressive trajectory structure. Neither line conditions on language subgoals, and neither studies how language interacts with the factorization choice.
Research Program
We will build a single policy family with three interchangeable parts. A shared transformer encoder reads the observation history and goal. An optional subgoal head predicts a short subgoal string; in BabyAI and Crafting the subgoal vocabulary is small, so a few AR tokens is enough and costs little. A chunk decoder emits $H$ discrete actions, run in one of four modes. The modes are fully AR over the chunk, one-shot parallel, iterative refinement with $K$ passes in the style of Lee et al., and masked discrete diffusion with $K$ steps. The subgoal is either absent, predicted by the model, or replaced by a control signal of matched vocabulary size, described below. During training the subgoal comes from the demonstration's annotation. At test time it is predicted, and the chunk is executed with receding-horizon replanning.
The analysis side estimates $\mathrm{TC}$ with and without subgoals for every environment and chunk length. It then asks whether $\Delta_{\mathrm{TC}}$, computed from data alone before any policy is trained, predicts the measured language gain across settings.
BabyAI is useful because its demonstrations come from a scripted planner, which lets us check coherence directly. We call a chunk "mode-mixed" if it is not a prefix of any shortest path consistent with some valid next subgoal. This gives a direct measure of the failure mode, independent of final success.
Baselines
The naive reference point is a language-free AR policy that predicts one action at a time. Its reported numbers are 73.6% success on BabyAI BossLevel as the caption-free baseline in Think Before You Act. The original BabyAI agent reaches 77% on BossLevel only with one million demonstrations. We retrain this baseline on our demonstration budgets with the same encoder, so it anchors every comparison.
The strongest existing language-using method is the interleaved caption-action policy of Think Before You Act, which reports 85.2% success on BossLevel. We also include auxiliary instruction prediction from Improving Long-Horizon Imitation through Instruction Prediction, which reports 70% on BossLevel with 50,000 demonstrations. Both are run on identical demonstration sets (10k, 50k, 200k), the same encoder size, and the same hyperparameter search budget per method, fixed at an equal number of trials. Beating them does not mean higher absolute success, since they are sequential and pay per-step decoding. The bar is matching their success within confidence intervals at $H\geq 8$ while using $H$-fold fewer policy calls per episode.
For language-free parallel action decoding on BabyAI, no reported result was found on Lacuna, so we construct that baseline ourselves. It is the same chunk decoder with no subgoal, in both one-shot and iterative-refinement modes.
The ablations that isolate the key ingredient replace the true subgoal with three controls. The first is a shuffled subgoal drawn from another episode. The second is a learned discrete latent code with the same vocabulary size, trained end to end in the style of a vector-quantized plan. The third is a subgoal produced by a sequence-level distillation step. The claim survives only if true subgoals reduce mode mixing and raise success beyond these controls in the regimes where $\Delta_{\mathrm{TC}}$ is large.
Why not just…?
Why not just keep the policy autoregressive, since BabyAI actions are cheap to decode? For short captions and single-step actions, the latency argument is weak, and we do not rest the proposal on it. The scientific claim is that language's value depends on the decoding factorization, and an AR-only study cannot test that. The practical case is chunked control, where fewer policy calls matter. If AR chunk decoding with language matches the parallel variants on both success and calls per episode, we will report that parallel decoding buys nothing in this domain.
Why not just learn a discrete latent plan instead of using language? This is the most serious threat. Mode selection only needs some committed variable, and a learned code might supply it. Our expectation is that language wins at low data because it is a supervised label for the mode, whereas a learned code must discover modes from scarce demonstrations. The two should converge at large data. The matched-vocabulary latent ablation across the 10k–200k sweep settles this. If the latent matches language at every data scale, the conclusion becomes that "instructional utility" is really a mode-labeling effect, and we will state it that way.
Why not just distill from an AR teacher, as parallel translation does? Sequence-level distillation in Non-Autoregressive Neural Machine Translation collapses multimodal targets into one mode without any plan variable. Imitation Learning for Non-Autoregressive Neural Machine Translation shows teacher-guided training of parallel decoders in text. We will distill a language-free parallel policy from the AR chunk policy. The risk to distillation is specific to control: collapsing to one mode is fine in translation but can remove the agent's ability to switch routes after a perturbation. We test this with perturbed-start episodes.
Why not just use speculative decoding for speed? DFlash keeps an AR target distribution and so sidesteps mode mixing entirely. As a latency tool it is a legitimate alternative, and we will report its calls-per-episode. It does not answer the question posed here, which is about how conditioning information interacts with independent per-position prediction.
Proposed Experiments
The main grid crosses environments (BabyAI GoToLocal, PutNextLocal, BossLevel; Crafting at 1 to 5 steps; ALFRED as the redundancy control), chunk lengths $H\in\{1,4,8,16\}$, the four decoding modes, subgoal conditions, and demonstration budgets. We report success rate, the fraction of mode-mixed chunks, policy forward passes and wall-clock time per episode, and the estimated $\mathrm{TC}$ with and without subgoals.
The first decisive check is the interaction. The language gain, measured as success with subgoal minus success without, should increase with $H$ for parallel decoders and stay roughly flat for AR decoders. If it is flat for both, the core claim is wrong.
The second check is prediction. Across all environment and $H$ cells, $\Delta_{\mathrm{TC}}$ estimated from demonstrations alone should rank-correlate with the observed language gain for parallel policies. It should be near zero on ALFRED and on GoToLocal, which matches the redundancy findings of Hejna et al..
The third check is mechanism. On BabyAI, the reduction in mode-mixed chunks from true subgoals should exceed that from shuffled subgoals and account for most of the success gain. A gain that appears without any drop in mode mixing would mean language is helping through representation shaping, the explanation Hejna et al. offer, rather than mode selection.
The fourth check is the saturation regime. On 5-step Crafting tasks, where agents lack the required skills, we expect $\Delta_{\mathrm{TC}}$ to stay large while the gain shrinks. That would separate "language could select a mode" from "the policy can execute the mode," and would locate the upper bound of instructional utility in execution rather than coordination.
References
- Improving Long-Horizon Imitation through Instruction Prediction
- Think Before You Act: Unified Policy for Interleaving Language Reasoning with Actions
- Teaching Embodied Reinforcement Learning Agents: Informativeness and Diversity of Language Use
- Non-Autoregressive Neural Machine Translation
- Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement
- Thinking Clearly, Talking Fast: Concept-Guided Non-Autoregressive Generation for Open-Domain Dialogue Systems
- Promises and Pitfalls of Generative Masked Language Modeling: Theoretical Framework and Practical Guidelines
- Why Diffusion Language Models Struggle with Truly Parallel (Non-Autoregressive) Decoding?
- Bringing Stability to Diffusion: Decomposing and Reducing Variance of Training Masked Diffusion Models
- Goal-Conditioned Imitation Learning using Score-based Diffusion Policies
- Chain-of-Action: Trajectory Autoregressive Modeling for Robotic Manipulation
- CDP: Towards Robust Autoregressive Visuomotor Policy Learning via Causal Diffusion
- BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning
- Imitation Learning for Non-Autoregressive Neural Machine Translation
- DFlash: Block Diffusion for Flash Speculative Decoding
- Improving Long-Horizon Imitation Through Language Prediction