Robots that combine a large vision-language model for planning with a small, fast motor policy have a scheduling problem. The large model is slow and expensive, so it should run only when it is needed. In a team of robots, most of the moments that need slow deliberation come from other robots: a handoff that is about to fail, two arms reaching for the same region, or a partner whose plan has quietly changed. We propose to detect those moments with the fast motor policy itself. Each robot's fast policy attends over short "intent" messages from its peers. At every control step it asks a counterfactual question: would my next action change if I ignored peer $j$? If the answer is yes for some peer, the robot calls its slow reasoner. It gives the reasoner only the messages from the peers that mattered, and it sizes the reasoning budget to how large the disagreement is.
The same cheap signal does three jobs. It decides when to think, what to read, and roughly how long to think. Prior systems handle these separately or not at all. Single-robot orchestrators learn when to think but have no peers (RARRL). Multi-robot language systems talk at every turn with no filtering (RoCo). Communication filters prune messages but treat the receiver's reasoning as a fixed cost (DCC, KVComm). If the idea works, collaborative robots could spend far fewer large-model tokens and keep their fast control loop running without losing task success.
Core Claim
In hierarchical multi-robot manipulation, peer influence is a better trigger for slow reasoning than generic triggers. By peer influence we mean how much a fast policy's action distribution shifts when one peer's message is masked out. The generic triggers are a fixed schedule, the robot's own action uncertainty, spatial proximity, or a state-based learned gate. Restricting the reasoner's context to the influential peers should also preserve success at lower token cost than giving it every message.
Concretely, robot $i$ runs a fast policy $\pi_i$ that sees its observation $o_i$ and the set of peer intent messages $m_{-i}$. We define the influence of peer $j$ at time $t$ as
$$I_{ij}(t) = D_{\mathrm{KL}}\big(\pi_i(\cdot \mid o_i, m_{-i}) \,\|\, \pi_i(\cdot \mid o_i, m_{-i\setminus j})\big),$$
where $m_{-i\setminus j}$ is the message set with peer $j$'s message masked out. The robot calls its reasoner when $\max_j I_{ij}(t)$ exceeds a threshold $\tau$. The reasoner's context contains only messages from peers with $I_{ij}(t) > \tau$. A token budget tier is chosen from $\sum_j I_{ij}(t)$.
This extends the "masking trick" of DCC in three ways. DCC compares a grid agent's planned action with and without a neighbor, and uses a binary result to decide whom to talk to. We instead make the check continuous, run it inside a hierarchical controller, and use it to schedule and size language-model reasoning rather than message exchange.
Research Program
What each robot broadcasts. Each robot sends a compact intent message rather than dialogue or attention caches. The message contains its current subgoal embedding, its short-horizon planned end-effector trajectory, and its robot identity. Robot identity matters for heterogeneous arms such as the UR5 and UR10 pair.
We avoid KV-cache sharing for two reasons. KVComm transmits attention key-value matrices between LLMs and requires identical model architectures on both sides. It also still carries dense payloads. Both properties are poor fits for heterogeneous physical teams.
Training the fast policy. The fast policy is a small transformer with one cross-attention layer over peer messages. We train it by imitation on multi-robot demonstrations. During training we randomly drop peer messages, so that the masked forward passes used at test time are in-distribution. Without this dropout, the influence score would partly measure how the policy reacts to unfamiliar inputs rather than to the missing peer.
The slow reasoner. The reasoner is a roughly 4B-parameter vision-language model, the scale used by R³. R³ trains such a model with RL to reason in free text before emitting short-horizon instructions, and it observes that reasoning length grows on hard cases. We reuse that setup, but add an explicit budget.
Budget tiers. Following HBPO, which trains with several token-budget tiers and piecewise rewards to avoid the collapse caused by flat length penalties, the reasoner is fine-tuned to respect three budget tiers. Its reward is task success minus a within-tier cost.
Asynchronous execution. The fast loop never waits for the reasoner. It keeps acting on the latest subgoal, as in HiRT. HiRT runs a 7B planner asynchronously beside a 10 Hz policy and reports that success rose from 48% to 75% under moving objects. The difference is that HiRT's planner runs at a fixed rate, while ours runs on demand.
Choosing the threshold. The threshold $\tau$ is the only new hyperparameter. We sweep it to trace a curve of success against tokens, instead of reporting a single tuned operating point. Masking requires $N-1$ extra forward passes of the small policy per step. For teams of two to four arms, this is negligible next to a single reasoner call.
Baselines
Naive reference points. The first is "always think." Every robot calls the reasoner at every high-level decision point with all peer messages in context, which matches RoCo's protocol of full dialogue at every coordination turn. The second is "never think." The fast policy follows only its initial subgoal assignment, which shows how much reasoning is worth at all.
Fixed-period replanning. The reasoner runs every $K$ seconds, with $K$ chosen so that its call count matches ours.
Strongest existing method. The strongest competitor is a per-robot RARRL orchestrator. RARRL is a PPO policy that observes state, execution history and remaining budget, and chooses between acting and thinking at a chosen token budget. We give it the full set of peer messages as input. It is the only existing method that learns when and how much to think inside an embodied hierarchy, so it is the natural rival.
Its reported results are for a single robot. It cut LLM inference time by over 60% relative to always-on reasoning on ALFRED at similar or better success. It used 580 tokens per episode versus 4,100 for full reasoning on its abstract MDPs. No reported result found on Lacuna for RARRL on any multi-robot benchmark.
For fairness, RARRL's orchestrator is trained on the same demonstrations and simulator episodes as our fast policy. It uses the same reasoner and the same budget tiers, and it gets a tuning budget matching our $\tau$ sweep. Beating it means a higher area under the curve of success against tokens on RoCoBench and LEMMA. At matched success, it also means at least 30% fewer tokens per episode.
Task-performance reference on LEMMA. The hierarchical Episodic Transformer + CLIPort system in LEMMA reports 87% success on "Pass" and about 73% on "Hook&Stack" with ground-truth robot allocation. Success drops sharply without ground-truth allocation. On RoCoBench, no reported per-task success figures found on Lacuna; we therefore rerun RoCo ourselves.
Ablations isolating each ingredient. Four ablations separate the pieces of the method:
- Influence triggering with full context tests whether filtering matters.
- Influence-filtered context with a fixed budget tests whether sizing the budget matters.
- Random triggering at the same call rate tests whether timing matters.
- Replacing the masked-peer divergence with the fast policy's own action entropy tests whether the peer counterfactual matters beyond generic uncertainty.
Why not just…?
Why not just trigger on the robot's own uncertainty? This is the most serious threat. A fast policy whose action entropy spikes may be facing exactly the peer conflicts we care about. The two signals should diverge when a policy is confidently wrong because of a peer: for example, it is confident about a grasp that a partner is about to occlude, or about a handoff pose the receiving arm cannot reach. LEMMA's failures without ground-truth allocation suggest such cases exist, but that is not proof. The decisive-check experiment below measures both triggers against an oracle. If entropy matches influence, we will report that the counterfactual adds nothing. The contribution then shrinks to context filtering.
Why not just use proximity? With two to four arms, "think when another arm is within $d$ centimeters" may track influence closely. DCC found that nearest-neighbor communication used about 19,490 pairs versus 1,811 for causal selection, with lower success, but that was 64 agents on a grid. For manipulation we must show the gap ourselves. RoCoBench varies workspace overlap and observation asymmetry, which lets us test the prediction directly. Proximity should do well on high-overlap tasks and fail on asymmetric-observation and handoff tasks, where a distant peer's changed plan matters. If proximity matches us everywhere, the idea does not justify its machinery.
Why not just let a learned orchestrator decide from the full state? RARRL-style gating could in principle learn peer influence implicitly from raw peer messages. Our bet is sample efficiency. The counterfactual signal is available at every step without any RL on the gate, while a PPO orchestrator must discover from sparse team reward which peer configurations need reasoning. The matched-data comparison settles this. We additionally report both methods at reduced training-episode counts.
Why not just cap the reasoner's length? HBPO shows that flat length penalties collapse reasoning on hard problems. Self-Braking Tuning, which teaches models to stop once an answer is verified, notes that hard token cutoffs truncate derivations mid-stream. We include a fixed-cap reasoner as a variant of the fixed-budget ablation.
Proposed Experiments
Environments. We run RoCoBench's six collaborative tasks and the LEMMA Isaac Sim suite with heterogeneous UR5/UR10 arms and 6,400 demonstrations. On LEMMA we focus on "Pass" and "Hook&Stack," where allocation and reachability dominate. We add a dynamic variant of each, in which a peer's target object is displaced mid-episode. This follows HiRT's moving-object protocol and tests whether the fast loop stays reactive while the reasoner deliberates.
Metrics. We report task success, reasoner tokens per episode, reasoner calls per episode, wall-clock time to completion, and the realized fast-loop rate. We also count coordination failures: collisions, dropped handoffs, and duplicated subtasks.
Decisive check. We sample states from rollouts of the "never think" policy. At each state we fork two continuations, one with a reasoner call injected there and one without. We label the state "reasoning helped" if only the forked call rescues the episode.
We then score each candidate trigger against these labels by AUROC, which measures how well a signal ranks helpful states above unhelpful ones. The candidates are peer influence, own-action entropy, proximity, and RARRL's learned gate probability. The core claim predicts that peer influence has the highest AUROC. The gap should be largest on asymmetric-observation and heterogeneous-arm tasks.
Secondary check. On the same states, we compare reasoner outputs given filtered versus full context. This tests whether dropping low-influence peers ever removes information the reasoner needed. A failure here would reverse the filtering half of the claim.
References
- When Should a Robot Think? Resource-Aware Reasoning via Reinforcement Learning for Embodied Robotic Decision-Making
- RoCo: Dialectic Multi-Robot Collaboration with Large Language Models
- Learning Selective Communication for Multi-Agent Path Finding
- KVComm: Enabling Efficient LLM Communication through Selective KV Sharing
- $R^3$: Training Robots to Reason in Natural Language via Reinforcement Learning
- Hierarchical Budget Policy Optimization for Adaptive Reasoning
- HiRT: Enhancing Robotic Control with Hierarchical Robot Transformers
- LEMMA: Learning Language-Conditioned Multi-Robot Manipulation
- Let LRMs Break Free from Overthinking via Self-Braking Tuning