Revisable Referents: Constrained Robot Command Decoding That Does Not Lock In Pointing Errors

When a person tells a robot "pick up that one" while pointing, the robot has to turn a noisy gesture into a specific object and then into a valid command. A reliable way to keep the command valid is constrained decoding: a prefix tree (a trie, an index of every allowed string) masks the language model's next-token choices so it can only name objects and relations that exist in the robot's scene model. The catch is where the trie comes from. Every existing system builds it around an entry point that is chosen once, before decoding starts. In a multi-turn interaction, the object picked in turn one becomes that entry point for every later "it", "the one next to it", and "put it back". A slightly misread pointing ray therefore becomes a guaranteed-valid but wrong command, and the error stays in place for the rest of the conversation.

We propose to treat the referent — which physical object the person means — as a revisable belief rather than a committed answer. At each turn, the trie is rebuilt over a small connected set of candidate objects whose weights combine geometric cue evidence (pointing and gaze) with a text-plus-graph match between the utterance and the scene graph. The decoder keeps a lasting referent state that it can explicitly revise, and a physical action is only released when enough of the belief sits on one object. The claim is that this keeps the validity guarantee of trie decoding while stopping pointing errors from compounding over turns, and that neither hard committed tries nor unconstrained history-conditioned models can do both.

Core Claim

Trie-constrained decoding moves the risk of hallucination to the entry point. Graph-constrained Reasoning (GCR) builds a trie over the knowledge-graph neighborhood of text-linked seed entities and reaches a 100% faithful reasoning ratio on WebQSP and CWQ. Its authors also note that if the seed entity is linked wrongly, the trie is built on the wrong subgraph and reasoning fails completely. In embodied dialogue, the "entity linker" is a gesture interpreter such as the Touch-Line Transformer, and it has a known failure mode: it confuses objects that lie along the same line of sight. Our hypothesis is that in multi-turn interaction, hard entry-point commitment combined with conditioning on history turns small perception errors into persistent, valid-looking wrong commands. This is the structural version of the snowballing that Investigating and Mitigating the Multimodal Hallucination Snowballing documents for free-form vision-language models.

The mechanism has two parts. The first is a referent belief $b_t(v)$ over scene-graph nodes $v$, updated each turn $t$ as

$$b_t(v) \propto b_{t-1}(v)^{\lambda}\, g_t(v)\, s_t(v),$$

where $g_t(v)$ is the likelihood of node $v$ under the turn's pointing and gaze geometry (zero-to-one, flat when no cue is present), $s_t(v)$ is the match between the utterance and $v$'s text and graph neighborhood, and $\lambda \in [0,1]$ controls how quickly old evidence decays so that a fresh gesture can override history. The second part is that this belief enters decoding as a soft prior inside a hard trie. At the referent slot, only nodes in the candidate set $C_t=\{v : b_t(v) > \tau\}$ are legal, and each legal node's logit is shifted by $\log b_t(v)$:

$$\tilde z_{t,v} = z_{t,v} + \log b_t(v) \ \text{ if } v \in C_t, \qquad \tilde z_{t,v} = -\infty \ \text{ otherwise}.$$

Everything outside the referent slot — predicates, relations, command grammar — stays under the ordinary hard trie, so validity is still guaranteed. Only which valid object is chosen stays uncertain, and it can be revised.

Research Program

We build four components and test each one alone.

Scene model. The scene is a text-attributed scene graph: object nodes with short descriptions, and edges for spatial and support relations. For the semantic score $s_t$ we reuse the retrieval stage of G-Retriever, which embeds node and edge text, retrieves by similarity, and then solves a prize-collecting Steiner tree (a problem that picks a small connected subgraph maximizing total relevance "prizes" minus a size "cost"). We change one thing: the prizes are the current beliefs $b_t(v)$, not text similarity alone. The trie is then compiled from paths in that connected subgraph, following the KG-Trie construction in GCR. G-Retriever supplies the subgraph method and GCR supplies the trie method. What neither has is a candidate set driven by non-verbal cues and rebuilt every turn. G-Retriever also leaves its output decoding unconstrained, which we do not.

Cue likelihoods. The geometric term $g_t$ scores objects by angular distance from the eye-to-fingertip ray, which the Touch-Line Transformer showed beats the forearm ray. Where depth is available, we apply the speaker-frame rotation of REP so that "left of" is read from the human's side. Gaze adds a second cone, following the gaze-to-object mapping in Gaze-supported LLM for Bi-directional HRI. Because the two cones point along different axes, gaze can separate objects that lie on the same pointing line. This targets the Touch-Line failure mode directly.

Revisable state. The referent and slot state live in a persistent table that the decoder edits with trie-constrained operations: INSERT, DELETE, and a new REVISE that re-points an earlier referent. This extends Diable, which showed that editing a persistent state table is faster and less error-prone than regenerating the whole state each turn. Diable's own limitation is that a wrong INSERT persists until the user corrects it. REVISE is triggered automatically. When the Jensen–Shannon divergence between the history-carried belief $b_{t-1}^{\lambda}$ and the current-turn cue-only posterior $g_t s_t$ exceeds a threshold, the REVISE branch becomes legal in the trie. This borrows the divergence trigger of Residual Visual Decoding (RVD) from the snowballing paper, but applies it to a discrete referent distribution instead of blending free-form token distributions. LLM-as-RNN also self-heals a mutable memory, but it needs three model calls per step and its free-text memory breaks on small models. Our state is a constrained table updated within a single decoding pass.

Commit gate. Physical actions cannot be undone, so an action is emitted only when $\max_v b_t(v)$ passes a commit threshold. Below it, the decoder is limited to a clarification-question branch of the trie. This follows SafeDec's finding (SafeDec) that hard masking alone freezes robots while soft reweighting trades away safety. Here the soft part is limited to object choice, and the hard part covers validity and when to commit.

The output language is NatSGLD's Linear Temporal Logic (LTL) command format (a formal language for time-ordered goals such as "eventually hold the knife, then place it on the board") (NatSGLD), with atomic predicates grounded in scene-graph nodes. The backbone is a fine-tuned T5 or BART of the size used there.

Baselines

The naive reference is NatSGLD's unconstrained seq2seq mapping from speech and gesture to LTL, run on the same backbone and training data. NatSGLD reports a Jaccard similarity of 0.944 and a Spot score (logical equivalence) of 0.588 for BART, and a Jaccard of 0.961 for T5. The gap between high token overlap and low logical equivalence is the room constrained decoding should close on single-turn commands. A second naive point is an unconstrained model with the full dialogue history appended.

The strongest competitor on our actual question is a committed-referent pipeline: take the argmax referent from a Touch-Line/REP-style cue model, build a GCR-style hard trie around it, and edit a Diable-style table without REVISE. This is what someone would assemble today from the best published parts. It inherits GCR's 100% validity, and on single-turn embodied reference its grounding stage matches REP, which reports 45.7% [email protected] on YouRefIt. We will run it with the same cue extractors, scene graphs, backbone, and hyperparameter search budget as our method, so the only difference is commitment versus belief. We count it as beaten if, on multi-turn chains with the same 100% validity, our method achieves significantly higher chain-level task success, with the gap growing across turns and concentrated in chains whose first-turn referent was wrong. A single-turn gain alone would not count.

A second strong competitor is an unconstrained vision-language model decoded with RVD over the dialogue history. It targets the same snowballing, and the paper reports lifting LLaVA-1.5 from about 15% to 33% under hallucinated conversation contexts. We will give it the same scene rendering and cue overlays.

Ablations isolate the key ingredient. We remove REVISE while keeping the belief; replace the belief prior with a flat top-$k$ candidate set; set $\lambda=1$ (no decay) and $\lambda=0$ (no memory); drop gaze from $g_t$; and remove the commit gate.

For multi-turn embodied reference with gestures there is no reported result found on Lacuna. The multi-turn numbers will all be new, and all baselines will be run by us.

Why not just…?

Why not just re-resolve the referent from scratch each turn, with no memory at all? Statelessness cannot cascade, so this is a real threat. It fails only where a turn carries no gesture ("now put it on the shelf") and the referent must come from history. We will split results by whether a turn has a gesture. If stateless re-resolution matches our method on turns that lack a gesture, the belief carry-over is unnecessary and we will say so. The $\lambda=0$ ablation is exactly this system.

Why not just keep the top-$k$ referents in a hard trie (beam over entry points)? This removes single-point commitment and is the nearest cheap alternative. It differs from our method in two ways: it has no calibrated weights across candidates, and it has no mechanism to bring back a candidate that fell out of the beam in an earlier turn. The flat top-$k$ ablation tests the first difference and the no-REVISE ablation tests the second. If top-$k$ with $k=3$ recovers most of the gain, the contribution shrinks to "don't commit early", and we would report it that way.

Why not just prompt a frontier model with the serialized scene and full history? The file's evidence goes against it. GCR reports that about 33% of reasoning paths generated by strong baselines are invalid and that faithful reasoning falls to 48–62% without constraints. The snowballing paper reports at least a 31% drop and a nearly 100% flip rate once a wrong premise enters the history. We still run the RVD-decoded vision-language model and a history-prompted baseline, because embodied scenes are small and a small scene might suffer less.

Why not just filter invalid commands after generation? SafeDec reports that post-hoc filtering drops task success by 10–15% through repeated invalid samples. Filtering also does nothing about valid-but-wrong referents, which is the failure this proposal targets.

Proposed Experiments

No existing benchmark is multi-turn, embodied, and gesture-grounded, so we build chains from existing resources and say so plainly. From EQA-MX's synthetic scenes (EQA-MX), where the authors show object grounding falls from about 68% to 26% without gaze and pointing, we generate 3–6 turn chains over one scene. Only the first turn is guaranteed to carry gestures, and later turns use pronouns and relations. From NatSGLD's 1,143 kitchen commands we assemble task sequences with LTL targets, which lets us measure the Spot score per turn and per chain. YouRefIt serves only to calibrate $g_t$ and to confirm that our cue module reaches REP-level single-turn precision, so that later gains cannot be blamed on a weak baseline grounder.

The decisive check is a controlled cascade test in the style of MMHalSnowball. In chains where the first turn is correct, we inject calibrated perception errors: wrist jitter, gaze noise, and depth-rotation error. We also deliberately place the wrong object on the same pointing line. We then measure final-turn accuracy, turn-of-first-failure curves, the recovery rate (the fraction of chains with a wrong first referent that end correct), the wrong-physical-action rate, and the clarification rate. Our method should show flat or slowly falling accuracy with depth where the committed pipeline falls steeply, with command validity equal (100%) for both. Two results would falsify the central claim: the committed pipeline's degradation curve matches ours, or the commit gate reduces wrong actions only by asking so many clarifying questions that the task goes nowhere. We also report per-turn decoding latency against the committed pipeline and LLM-as-RNN, since real-time interaction is part of the motivation.