Deciding Before Speaking: Coherent Parallel Generation of Multi-Channel Operator Alerts

When a semi-autonomous drone or surgical robot hits trouble, it must tell its human operator quickly. The best alerts spread information across senses: a spoken sentence, a highlighted icon, sometimes a vibration. Human-factors studies show that this bi-modal delivery beats single-channel alerts. They also show that the channels do not need to be rigidly time-locked. Fast generative models are an obvious way to produce such alerts on the fly. The fastest of them, parallel decoders that write many output tokens at once, rely on a shortcut: they treat simultaneously generated tokens as independent of one another given the input. Inside one sentence this shortcut causes stutters. Across channels it can cause something worse. The speech can say "rotor failure, land now" while the icon lights up "battery low, return home," because each channel picked a different plausible story.

This note proposes a way to get alerts that agree across channels without giving up parallel speed. The model first decodes a handful of "decision tokens," the few output positions that carry nearly all the information the channels must share. It then decodes the rest of every channel fully in parallel. We will test the proposal from two sides. On the generation side, we check whether conditioning on those few tokens really removes cross-channel dependence. On the human side, we run an operator study that asks whether cross-channel contradictions hurt take-over decisions more than the within-sentence glitches that current parallel decoding research works to remove.

Core Claim

A theory result motivates the claim. On the Learning of Non-Autoregressive Transformers proves that any model which generates tokens independently pays an error floor set by the conditional total correlation of the target. This is the gap between the sum of per-token uncertainties and the joint uncertainty, and it measures how much the tokens depend on each other. Grouping the output into channels $c$ (speech, icon slots, haptic slots) with contents $X_c$ and scene input $Y$, the relevant quantity for alerts is the cross-channel part:

$$C_{\text{cross}}(X \mid Y) = \sum_{c} H(X_c \mid Y) - H(X \mid Y),$$

where $H$ is conditional entropy. Our claim is that for operator alerts $C_{\text{cross}}$ is concentrated. It is mostly the operator's uncertainty about which failure and which action, and that uncertainty is resolved by a small set $D$ of positions, such as the failure-type noun and the prescribed maneuver in speech, plus the one active icon. Once $D$ is fixed, the residual $C_{\text{cross}}(X \mid Y, D)$ should be close to zero. Huang et al.'s bound then says the remaining tokens of all channels can be decoded in parallel with little loss. The new contribution is not a new sampler in general. It is three things: identifying and exploiting this structure for multi-stream outputs, a way to find $D$ by measurement rather than by hand, and evidence about whether the resulting coherence matters to people under time pressure.

Research Program

We fine-tune a discrete diffusion multimodal model on alert generation. The model will be Dimple, a vision-language model that generates by iteratively unmasking tokens and is released with confidence-thresholded parallel decoding and fixed "structure prior" format tokens. The output is one sequence with pinned channel delimiters: a spoken sentence, a set of icon-slot tokens, and a short haptic-pattern token string. Its input is a rendered cockpit view plus a telemetry text summary. Dimple's structure priors are a natural fit for pinning channel boundaries. Its confident decoding, however, unmasks whatever tokens are confident, with no regard for which channel they belong to.

No multi-channel alert corpus exists in the file, so we build one. Seeds are the 20 failure scenarios and five interface conditions of the AirSim drone take-over study in The Design of Informative Take-Over Requests for Semi-Autonomous Cyber-Physical Systems. That study used pre-scripted audio and hardcoded icon rules. We extend it with compound and held-out failures, for example rotor degradation combined with a GPS dropout. An autoregressive teacher model produces candidate alerts, and human annotators validate them for cross-channel agreement. We keep several valid alerts per scenario on purpose, because multimodality of the target is exactly what makes $C_{\text{cross}}$ large.

Decision tokens are found by intervention. On training data, we fix each candidate position to its sampled value and measure the average shift (KL divergence) it induces in the other channels' marginals. Positions whose influence is high form $D$. A small head on the frozen backbone learns to predict per-position cross-channel influence at inference time. Decoding then runs in two phases. It spends one to three steps committing the top-influence positions, then unmasks the remainder with ordinary confident decoding across all channels at once. We will also estimate $C_{\text{cross}}$ before and after conditioning on $D$, using an autoregressive scorer on the corpus, to test the concentration hypothesis directly rather than only through downstream metrics.

Baselines

The naive reference point is Dimple's own confident decoding applied to the concatenated multi-channel sequence with no cross-channel awareness. This is what a practitioner gets today, and it fixes the speed end of the comparison.

The quality ceiling is autoregressive decoding of the same fine-tuned model, which respects all dependencies at the cost of one step per token. Beating it on coherence is not expected. The target is matching its contradiction rate within statistical noise at a fraction of its forward passes.

The strongest existing parallel competitors are two, and both are run on the identical fine-tuned backbone, data, and step budgets. The first is DAPD. It builds a dependency graph from attention weights among masked positions and unmasks only mutually weakly-attending sets. Its authors report more than 1.8x speedup over competing parallel strategies and 3.8x over step-by-step unmasking on LLaDA. It is the closest competitor because it already tries to unmask only nearly independent tokens. It uses attention as a proxy for dependence and makes no distinction between within-channel and cross-channel edges. The second is CoDD, which multiplies the Transformer's per-token outputs by a small tractable hidden-Markov-style model to restore inter-token coupling. On Dream-7B at 64 steps it raised GSM8K accuracy from 34.0% to 56.4% with about 3 GPU hours of training. We train CoDD's circuit on our corpus with the same compute budget as our influence head. Its fixed 32-token window may span channels or may not, and we will report both placements. There is no reported result on Lacuna for any of these methods on multi-channel alert generation.

Two ablations isolate the key ingredient. The first commits the same number of tokens in phase one, but chooses them by confidence or at random. This tests whether which tokens are committed matters or merely that some are. The second uses DAPD's attention graph restricted to cross-channel edges. This tests whether intervention-measured influence beats attention as the dependence signal. Beating the competitors means a lower cross-channel contradiction rate at an equal number of forward passes, or equal contradiction at fewer passes, on held-out compound failures.

On the human side, the reference numbers come from the AirSim study: bi-modal scripted alerts reached 67.63% decision accuracy, compared with 56.86% for visual-only and 48.74% for a tone (Gundappa et al.). Scripted bi-modal alerts are the human-side ceiling our generated alerts should approach.

Why not just…?

Why not just classify the failure and fill channel templates? This is the most serious threat. A classifier followed by deterministic templates makes the channels agree by construction, and for the original 20 fixed scenarios it may well be as good as anything generative. Our design settles the question empirically. Templates are a baseline on both seen and held-out compound failures, and we expect them to degrade on the held-out set, where no template exists. There is a second reason to expect a gap: Gundappa et al. found that full natural sentences trended toward higher accuracy than fragments, and templated compound messages tend toward fragments. If templates match us on held-out failures, we will report that generation is unnecessary for this domain. The human-sensitivity study below would still stand as a contribution.

Why not just decode autoregressively and pace delivery in the interface? Pacing is indeed an interface problem, and we do not claim otherwise. The same study found that rigid word-icon synchronization slowed operators, so channels need not be released in lockstep. What autoregressive decoding costs is latency before the first complete alert, inside a 10-second take-over window. The multimodal speculative decoding diagnosis in Is Multimodal Speculative Decoding Ready for Diffusion-Based Parallel Drafting? also found that visual conditioning can cost up to 25x a drafting step. We therefore report end-to-end wall-clock time including image encoding, not just step counts. If the savings disappear under that accounting on realistic hardware, the autoregressive option wins, and we will say so.

Why not just apply DAPD or CoDD unchanged? They might suffice. That is why they are the main baselines rather than straw men. Our reason for doubt is that neither separates the dependencies that humans punish, those across channels, from those they tolerate. CoDD's short fixed window also leaves long-range links unmodeled, and an icon slot can sit far from the sentence's failure noun.

Proposed Experiments and Decisive Checks

The first decisive check needs no humans. We estimate $C_{\text{cross}}$ with and without conditioning on the discovered set $D$. If committing one to three positions does not remove most of the cross-channel correlation, the core claim fails regardless of downstream numbers.

The second check is automatic coherence. We parse the speech and icons into (failure type, maneuver) pairs and measure disagreement rate against forward passes and wall-clock time, across all baselines and ablations, on seen versus held-out compound scenarios.

The third check is a controlled human study in the AirSim setting that separates human sensitivity from generator quality. Participants receive alerts with artifacts injected at matched rates: within-speech stutters of the kind described in Promises and Pitfalls of Generative Masked Language Modeling, or cross-channel contradictions. We measure decision accuracy, reaction time, NASA Task Load Index workload ratings, and the "perception-action gap," meaning correct identification followed by a contradictory maneuver. If contradictions do not hurt more than stutters, the motivation for targeting cross-channel dependence weakens substantially.

The fourth check puts the best generators in front of participants against scripted bi-modal alerts. Success means approaching the scripted 67.63% accuracy on seen scenarios and beating templates on held-out ones. We add pupil and heart-rate recording as secondary workload measures. We treat them only as coarse indicators, since ADABase reports good binary separation (AUC 0.91) but poor fine-grained estimation across individuals.