Several language model agents can now pass internal numerical states to each other instead of writing text. This latent communication is fast, but it is applied uniformly: every query pays for the same exchange, whether the exchange helps or not. Sometimes it hurts. Latent messages between agents have made them worse than a single agent on ARC-Challenge, and a weak sending model drags down a strong receiving one. We propose that each agent summarize what it currently believes about the answer as a box, meaning an axis-aligned hyper-rectangle in a small shared space, where the box's size reflects how unsure the agent is. Comparing two agents' boxes then tells the receiver which of three cases it is in. If its own box already sits inside the sender's, the message is redundant and can be skipped. If the sender's box is small and inside the receiver's, the message is informative and worth fusing cheaply. If the boxes do not overlap, the agents disagree, and the system should spend more compute, for example by switching to an explicit text exchange or drawing more samples.
The idea in two sentences: give every agent a box-shaped "belief region" trained so that box overlap predicts whether two agents' answers agree. Then use the set relation between sender and receiver boxes (contained, containing, or disjoint) to decide per query and per round whether to skip communication, fuse latents, or escalate to costlier text deliberation. What is new is not boxes or latent messages on their own. It is using closed-form set operations between agents' uncertainty regions as a compute-allocation signal that needs no retraining per agent pair. We test the boxes directly against simpler geometries and against a directly supervised gate, and drop them if they lose.
Core Claim
Three bodies of work motivate the proposal.
Latent communication shows the channel is worth having, and that it misfires. Cache-to-Cache fuses one model's key-value cache (the stored attention keys and values from earlier tokens) into another frozen model through a learned gate. It reports 3.1–5.4% higher accuracy than text-to-text communication and a 2.5× average speedup, but it fuses on every query and degrades with a weak sharer. Enabling Agents to Communicate Entirely in Latent Space (Interlat) passes last-layer hidden states, compressed to a fixed 8-step prefix, and argues these carry a superposition of hypotheses. Nothing in it measures how many hypotheses are being carried.
On the Role of Learned Alignment Matrices in LatentMAS constrains the design. Its learned alignment matrices are near-identity, latent collaboration hurts on ARC, and on MBPP+ text communication reaches 89.0% against 73.3% for latent communication. A sensible system should therefore sometimes refuse latent messages and sometimes fall back to text. Our router is built to make exactly that choice.
Adaptive compute work supplies the method template. Adaptive Thinking (Sonata) reads prefill hidden states with a small network to predict self-consistency and cuts tokens by 20–60%. The LLM Already Knows predicts difficulty from the initial state with 93.09% ROC-AUC on ScienceQA. Learning How Hard to Think predicts the marginal gain of one more unit of compute and matches best-of-$k$ with half the samples. All three are single-agent, make one decision at prefill time, and output a scalar. A scalar per agent cannot express how two agents' beliefs relate to each other, and that relation is the quantity the communication decision depends on.
Box embeddings supply the geometry. Query2box represents conjunctive queries as box intersections. Probabilistic Box Embeddings for Uncertain Knowledge Graph Reasoning (BEUrRE) reads conditional probabilities off intersection-to-volume ratios and reports a 14% better squared-error calibration than point methods on CN15k. Smoothing the Geometry of Probabilistic Box Embeddings and the box-embeddings library fix the zero-gradient problem for disjoint boxes with Gumbel and softplus relaxations. None of these attaches boxes to the live hidden states of a reasoning model. We reuse their training machinery wholesale.
The claim we test has two parts. First, a box head on each agent's hidden state can be trained so that the conditional probability
$$P(R \mid S) = \frac{\mathrm{Vol}(B_R \cap B_S)}{\mathrm{Vol}(B_S)}$$
tracks how often answers sampled from receiver $R$ fall among answers sampled from sender $S$. Here $B_R$ and $B_S$ are the two agents' boxes and $\mathrm{Vol}$ is the smoothed volume. Second, because boxes are closed under intersection, heads trained independently against one shared space give a usable routing signal for agent pairs and agent triples never seen together in training. A gate trained directly on one specific pair cannot offer that.
Research Program
Box heads. Each agent gets a two-layer head that maps its final-layer state to a Gumbel box in a 32–64 dimensional space. The head is read at prefill time and again at the end of each communication round, which addresses the prefill-only limitation Sonata acknowledges. The base model stays frozen.
Training targets. For each training query we draw $k$ rollouts from every agent and record their final answers. Three losses use these rollouts. A volume loss ties each agent's box volume to its own answer disagreement rate, playing the role of Sonata's self-consistency target. A pairwise loss ties $P(R \mid S)$ to the empirical cross-agent agreement rate. A containment loss pushes the box toward the box of the agent's own rollouts whenever its answer set is a subset of another agent's.
The space is anchored by answer agreement, not by hidden-state alignment. This sidesteps the near-identity collapse reported for LatentMAS, and it works across model families whose hidden dimensions differ.
Router. At each round the receiver computes three quantities: containment of its box in the sender's, the reverse containment, and the overlap ratio. A small threshold policy maps these to one of three actions. "Skip" answers alone. "Fuse" runs one Cache-to-Cache fusion step. "Escalate" runs a text exchange or extra samples. The thresholds are tuned once on a development split for a target compute budget, so every result is reported as an accuracy-versus-compute curve rather than a single point.
Baselines
The naive reference points are the receiver answering alone and the sender answering alone. We also report always-fuse with Cache-to-Cache and always-text, which together bracket uniform allocation. Always-fuse is the strongest existing latent method on general reasoning: the file reports it beating text-to-text by 3.1–5.4% and individual models by 6.4–14.2% across MMLU-Redux, OpenBookQA, ARC-Challenge, and C-Eval. On code, the reference is TextMAS at 89.0% on MBPP+, against LatentMAS at 73.3% (On the Role of Learned Alignment Matrices in LatentMAS). No reported result was found on Lacuna for per-benchmark ARC-Challenge or GSM8K accuracy under latent communication, so we rerun those ourselves. All methods use the same model pairs, the same Cache-to-Cache fuser checkpoint, the same rollouts for supervision, and the same threshold-tuning budget on the same development split.
The strongest competitor on the routing question itself is a directly supervised gate. It is a Sonata-style two-layer network on the concatenated sender and receiver states, trained to predict which of skip, fuse, or escalate gives the correct answer at least cost.
We also adapt the marginal-reward allocator of Learning How Hard to Think. It predicts the gain from each costlier action and uses the paper's offline binning.
Three geometry ablations isolate the box. The first is a point plus a scalar confidence per agent, with the router using distance and the two confidences. The second is diagonal Gaussians, with overlap computed by Bhattacharyya coefficient and fusion by Gaussian product. The third is boxes with the pairwise loss removed.
An oracle router, which picks the cheapest correct action per query, gives the ceiling.
Beating always-fuse means matching its accuracy at no more than 60% of its compute, or exceeding it at equal compute. Beating the direct gate means matching it on trained pairs while clearly exceeding it on held-out pairs and three-agent settings.
Why not just…?
Why not just train a gate that predicts directly whether to communicate? On a fixed, known pair of agents this is a real threat, and it may well match or beat boxes. The gate learns exactly the decision we care about, with no geometric detour. Its weakness is structural. It is trained on joint features of a specific pair, so adding an agent or swapping a sharer means relabeling and retraining. Box heads are trained per agent against a shared answer-anchored space, and intersection of any number of boxes stays a box. The held-out-pair and three-agent experiments below settle the question. If the gate retrained cheaply per pair still wins there, the box contribution reduces to calibration analysis, and we will say so.
Why not just use diagonal Gaussians, which also have width and also fuse in closed form? Gaussian products are closed too, so closure alone does not favor boxes. What boxes add is exact emptiness: disjoint boxes give zero conditional probability, which maps directly onto "escalate." Boxes also come with a direct volume-as-probability reading, which BEUrRE showed calibrates better than point methods. Neither advantage is guaranteed to matter for routing. The Gaussian ablation is run with identical losses, and if it matches boxes, we report the routing protocol as the contribution and the geometry as interchangeable.
Why not just talk in text when unsure and stay silent otherwise, using token entropy as the trigger? Sonata reports that surface statistics like token entropy miss real reasoning difficulty. Entropy is also a per-agent scalar, so it cannot tell "both confident and agreeing" apart from "both confident and disagreeing." We include an entropy-threshold router as an extra baseline. The MBPP+ gap shows text deserves a place in the action set, which is why escalation exists. The question this proposal answers is when text is worth its roughly 113 seconds per example against 23–44 seconds for latent exchange.
Proposed Experiments
We use Cache-to-Cache's heterogeneous pairs (Qwen and LLaMA families) and LatentMAS's homogeneous Qwen3-4B agents. Evaluation covers ARC-Challenge, OpenBookQA, MMLU-Redux, and GSM8K for reasoning, and MBPP+ for code. Box heads are trained on rollouts from disjoint training splits.
The first decisive check is calibration. On held-out queries, the Spearman correlation between predicted $P(R \mid S)$ and empirical cross-agent agreement must exceed that of the point and Gaussian alternatives. If it does not, the router has no basis.
The second is the accuracy-versus-compute Pareto curve against all baselines. Compute is measured in both FLOPs and wall-clock time.
The third targets the documented failure cases. On ARC-Challenge, where uniform latent communication fell below a single agent, the router should recover at least single-agent accuracy. When a deliberately weak sharer is paired with a strong receiver, the router should mostly choose skip. On MBPP+, it should escalate to text often enough to approach the 89.0% text reference at substantially lower latency.
The fourth is transfer to agent combinations never seen in training. Train heads with pairs A–B and A–C, then evaluate on B–C and on the three-agent intersection, against a direct gate trained only on the original pairs.
Finally, mid-generation re-reading of boxes is compared against prefill-only routing, to test whether per-round allocation adds anything beyond Sonata-style one-shot decisions.
References
- Cache-to-Cache: Direct Semantic Communication Between Large Language Models
- Enabling Agents to Communicate Entirely in Latent Space
- On the Role of Learned Alignment Matrices in LatentMAS
- Adaptive Thinking: Large Language Models Know When to Think in Latent Space
- The LLM Already Knows: Estimating LLM-Perceived Question Difficulty via Hidden Representations
- Learning How Hard to Think: Input-Adaptive Allocation of LM Computation
- Query2box: Reasoning over Knowledge Graphs in Vector Space Using Box Embeddings
- Probabilistic Box Embeddings for Uncertain Knowledge Graph Reasoning
- Smoothing the Geometry of Probabilistic Box Embeddings
- Box Embeddings: An open-source library for representation learning using geometric structures