A robot that uses a vision-language model (a neural network that reads camera images and text and reasons about them) to decide where to go has a timing problem. The model is slow, taking two to five seconds per thought on a compact 1B-parameter backbone. A fast, small controller steers the robot ten times per second in the meantime. When the thought finally arrives, it describes a world that has already changed. The robot has moved, and so have the people around it. An instruction like "pass behind the person crossing from the left" refers to someone who is no longer where the model saw them. The best current system, TIC-VLA, repairs only half of this. It tells the fast controller how long the thought took and how far the robot itself moved, but it treats everyone else as frozen in place.
We propose to move the prediction into the slow reasoner itself. A domain-specialized compact vision-language model is trained so that its output describes the scene at the moment its guidance will be used, not the moment its input was captured. It does this for several possible arrival times at once, and the fast controller picks the entry matching the actual measured delay. Two older ideas supply the pieces. From predictive state estimation comes the forward model, an internal simulator that projects the system state across a known delay. From delay-adaptive reinforcement learning comes the "action packet", a small table of contingent outputs indexed by arrival time. Domain specialization is what makes a small model capable enough to learn this forecasting at all.
Core Claim
The claim is that stale semantic guidance is best corrected at the sender, not the receiver. The reasoner has the scene context needed to forecast other agents: who is walking toward whom, and who is about to turn. The fast controller has only low-level features and a timestamp. We train the reasoner with a "hindsight target." In simulation, a teacher model sees the frame at time $t+\Delta t$ and produces guidance. The student reasoner sees only frames up to $t$ plus a sampled delay $\Delta t$, and must reproduce that guidance. At deployment the student emits a compact packet of guidance latents for $K$ delay values drawn from the observed latency distribution of the edge hardware. Our prediction is that this beats receiver-side compensation. The gain should grow with two quantities: the length of the delay, and the density and speed of surrounding people. When delays are short or the scene is static, the gain should shrink to roughly zero. This pattern is what separates a real effect from a generic capacity increase.
Where This Sits in Prior Work
Motivating and closest work. TIC-VLA is the closest work and the baseline to beat. It decouples an InternVL3-1B "thinker" from a 10 Hz action expert, and passes the thinker's internal cached features together with the elapsed latency $\Delta t$ and the robot's own displacement $\Delta p$. It then trains the action expert with injected random delays of 0–10 s and fine-tunes with reinforcement learning. Its own ablation shows the value of latency awareness: dropping $\Delta p$ lowers success from 47% to 41%. Our proposal differs in where the correction happens and what gets predicted. TIC-VLA's thinker is time-agnostic and compensation is left to the receiver. Our thinker is trained to forecast the other agents over its own delay.
Latency reduction instead of compensation. Two works shrink or hide latency rather than correct for it. Fast-ThinkAct replaces written-out chains of reasoning with six continuous latent tokens, cutting a 3B model's latency from 5674 ms to 805 ms. It still runs at about 1.2 Hz and has no correction for motion during that window. QUART-Online compresses ten-step action chunks into discrete tokens so that an 8B model can drive a quadruped at 50 Hz. The chunks then execute open-loop. Both are complementary to our approach: they make $\Delta t$ smaller, and we make the remaining $\Delta t$ harmless. Fast-ThinkAct's latent tokens are also the natural format for our guidance packet, because they are small enough that sending $K$ of them is cheap.
Method sources. Three works supply the techniques we reuse. Adaptive Reinforcement Learning for Unobservable Random Delays shows that emitting a matrix of actions keyed by possible arrival times beats worst-case delay buffering under real WiFi jitter. There, HalfCheetah reaches roughly 8368 return against roughly 4159 for the next-best baseline. We lift that packet idea from low-level torques to semantic guidance, where its bandwidth cost matters less because latents are small. Predictor feedback in control theory, e.g. Event-Triggered Stabilization of Nonlinear Systems with Time-Varying Sensing and Actuation Delay, integrates the full state forward across the delay and resets the prediction when a new measurement arrives. We copy that reset structure: each new thought supersedes the old packet. Data Efficient Reinforcement Learning for Legged Robots plans from a state projected 20 steps ahead and trains with multi-step losses to control drift. That is the precedent for our hindsight target over a horizon.
Constraining work. Specializing Smaller Language Models towards Multi-Step Reasoning shows that concentrating a small model on one domain costs general ability. We accept that cost for a navigation-only thinker, but we measure whether adding the forecasting objective degrades plain scene understanding. LLMLight shows that a specialized 0.5B model can beat Llama2-70B on traffic control. This supports the bet that a 1B thinker can learn short-horizon crowd forecasting, but it also assumes instant observation, which we do not.
Research Program
Thinker. We start from the TIC-VLA recipe with an InternVL3-1B thinker in the DynaNav Isaac Sim suite, so that every difference is attributable.
Hindsight targets. We generate these by running the frozen, supervised-fine-tuned thinker on the future frame at $t+\Delta t$ for $\Delta t$ sampled over 0–5 s. The student is then trained on frames up to $t$, with a learned embedding of $\Delta t$, to match the teacher's guidance latents and reasoning output. The objective is:
$$\mathcal{L} = \mathbb{E}_{\Delta t \sim p_{\text{lat}}}\, \big\| g_\theta(o_{\le t}, \Delta t) - g_{\text{teacher}}(o_{t+\Delta t}) \big\|^2 + \lambda\, \mathcal{L}_{\text{SFT}}$$
The terms are defined as follows. $g_\theta$ is the student's guidance latent. $o_{\le t}$ are the observations available when thinking starts. $p_{\text{lat}}$ is the measured latency distribution of the target hardware. $\mathcal{L}_{\text{SFT}}$ is the original supervised reasoning loss, weighted by $\lambda$ to limit drift.
Packets at deployment. The thinker runs the $\Delta t$-conditioned head at $K$ quantiles of $p_{\text{lat}}$, computed in one batched pass over a shared prefix. It then ships the $K$ latents. The action expert interpolates between the two entries that bracket the actual arrival time, and continues to use $\Delta p$ and its own live sensing. The expert is fine-tuned with the same delay injection and reinforcement learning stage as TIC-VLA.
Analyses. We probe what the forecast contains by decoding predicted pedestrian positions from the latents. We also measure the added cost of $K$ heads on a Jetson Orin NX.
Baselines
Naive reference. The do-nothing reference is TIC-VLA's architecture with $\Delta t$ and $\Delta p$ zeroed at the interface, i.e. stale guidance consumed as if fresh. It shows how much any compensation matters.
Strongest existing method. The main competitor is full TIC-VLA, which reports a 55.29% success rate on DynaNav's 85 scenarios. For comparison, it reports 32.94% for MobileVLA and 30.59% for DualVLN, and 0.75 success on a Jetson Orin NX in the real world. It is the strongest competitor because it already has latency-aware training, delay injection and reinforcement learning, and differs from us only in where compensation lives. We will retrain it ourselves with identical data, delay distribution, reinforcement learning steps and seeds. We will report both our reproduction and the published 55.29%, and judge ourselves against the reproduction. Beating it means a statistically reliable gain in success rate and collision rate across seeds. The gain must be concentrated in scenarios with long delays and dense crowds, and must not come with losses in static scenes.
Fast-reasoning baseline. A Fast-ThinkAct-style latent thinker without any hindsight target tests whether shrinking the delay alone is enough.
Key-ingredient ablations. Two ablations isolate what is new. The first keeps the $\Delta t$-conditioned student but trains it on present-frame rather than future-frame targets. This separates the forecasting target from mere delay conditioning. The second replaces the $K$-entry packet with a single forecast at the mean delay, which tests whether the contingent packet matters under heavy-tailed jitter.
Why not just…?
Why not just let the fast controller handle moving people, since it sees live sensor data at 10 Hz? This is the most serious threat. TIC-VLA's action expert already receives current observations and could learn to reinterpret stale guidance. The fast controller sees where people are now, but it lacks the thinker's semantic plan. That plan, such as which gap to aim for or whom to yield to, may be invalidated by crowd motion in ways that local features cannot repair. We cannot cite evidence settling this, so the TIC-VLA comparison, stratified by crowd density, is the direct test. If the gain vanishes there, the thesis is wrong.
Why not just extrapolate pedestrians at constant velocity and feed the thinker a predicted future image or annotated map? This cheap predict-then-think pipeline could capture most of the benefit in simple crossings. We include it as a baseline, rendering constant-velocity forecasts of tracked agents onto the thinker's input. Our model should win only where people turn, stop or react to each other, so we report results split by trajectory curvature. A tie in straight-line scenes would be expected and honest.
Why not just make the thinker faster? Fast-ThinkAct cuts latency sevenfold, but 805 ms at walking speed is still far from zero. Edge throttling also produces tails that no average speedup removes, as seen in TIC-VLA's Jetson result. Our sweep over injected delay shows where speedup suffices. If the latent-speedup baseline matches us across all delays down to realistic Jetson latencies, forecasting is unnecessary. We will say so.
Why not just buffer to the worst-case delay? ACDA's WiFi results show that fixed worst-case buffering roughly halves return relative to adaptive handling. We nonetheless include a fixed-$T_{\max}$ variant of our thinker: a single forecast at the 95th-percentile delay.
Proposed Experiments and Decisive Checks
Primary evaluation. This is DynaNav across its Hospital, Office, Warehouse and Outdoor environments. We inject delays drawn from three sources: uniform 0–5 s, a heavy-tailed distribution, and traces recorded from the Jetson Orin NX. We report success rate, collision rate, time to goal, and a forecast-quality probe. The probe is the error of pedestrian positions decoded from the guidance latents against ground truth at arrival time.
Real-world data. SCAND supplies real human-crowd trajectories. We use it both to pretrain the forecasting head on real motion and to test open-loop forecast quality off simulation. The reported 0.26 Hausdorff distance of its behavior-cloning planner serves only as context, not as a comparable number.
Real hardware. A small Unitree Go2 study compares against TIC-VLA's reported 0.75 on the Jetson.
The decisive check. We plot the gain over reproduced TIC-VLA as a function of injected delay and crowd speed. The hypothesis predicts a gain near zero at short delay or in static scenes, rising with both. A flat gain would point to a capacity artifact. No gain at all would mean the receiver-side controller was already enough.
References
- TIC-VLA: A Think-in-Control Vision-Language-Action Model for Robot Navigation in Dynamic Environments
- Fast-ThinkAct: Efficient Vision-Language-Action Reasoning via Verbalizable Latent Planning
- Quart-Online: Latency-Free Multimodal Large Language Model for Quadruped Robot Learning
- Adaptive Reinforcement Learning for Unobservable Random Delays
- Event-Triggered Stabilization of Nonlinear Systems with Time-Varying Sensing and Actuation Delay
- Data Efficient Reinforcement Learning for Legged Robots
- Specializing Smaller Language Models towards Multi-Step Reasoning
- LLMLight: Large Language Models as Traffic Signal Control Agents
- Socially CompliAnt Navigation Dataset (SCAND): A Large-Scale Dataset of Demonstrations for Social Navigation