Latent Bridge: Asking for Help Without Words

Qichao Wang, Tingyu Qu, Pengcheng Wu, Yiran Zhong, Feida Zhu, Zhiqi Shen, Peilin Zhao, Steven HoiarXiv 2026 · arXiv

TL;DR

POND (Prefill-Only, No-Decode) is a new framework that allows a large "sender" model to guide a small "receiver" model by sharing internal latent representations instead of generated text. By skipping the slow, token-by-token decoding of the large model and using only its fast parallel "prefill" pass, POND achieves significant accuracy gains in reasoning and coding while drastically reducing inference latency.

What Problem Does This Paper Address?

In the current landscape of AI, we often want the "brains" of a frontier-scale model (like GPT-4 or Qwen-Max) to assist a smaller, more efficient model running on edge hardware. Traditionally, this collaboration happens via natural language: the big model writes a hint or a chain-of-thought, and the small model reads it.

The bottleneck is latency. Large models are notoriously slow at autoregressive decoding—generating text one token at a time. However, they are incredibly efficient at "prefilling," where they process all input tokens in parallel. The authors ask a compelling question: Can we transfer the intelligence of a large model's prefill representations directly to a small model, skipping the text generation step entirely?

Methods

The authors introduce POND, which decouples the large model's understanding from its writing.

  1. The Latent Bridge: A frozen large "sender" model prefills the input prompt. A lightweight, trainable "bridge" (an MLP) compresses the sender's final-layer hidden states into a compact latent message.
  2. KV Adapters: This message is sent to the small "receiver" model. Layer-specific adapters project these latents into additional Attention Keys and Values (KV). Essentially, the small model "sees" the large model's thoughts as extra context in its own attention mechanism.
  3. Gated Joint Attention: The receiver uses a learned gate to balance its own internal representations with the guidance provided by the sender.
  4. Periodic Review: For long responses, the initial guidance can "go stale." POND allows the sender to periodically re-read the receiver's partial output. Because of "prefix determinism," the sender only needs to prefill the new tokens, updating the latent memory without recomputing everything.
Figure 1: Overview of POND. A frozen sender prefills the input, and a trainable bridge compresses its last layer hidden states into a shared latent message, which layer-specific adapters project into KV entries for the receiver. The receiver, fine-tuned jointly with the bridge, generates the full response. At review boundaries, the sender prefills the newly generated segment and the resulting entries are appended to the latent memory, leaving the receiver’s KV cache untouched.

Datasets

The researchers evaluated POND using a Qwen3.5-4B receiver paired with various frontier senders (Qwen3.8-Max, GLM-5.3, DeepSeek-V4-Flash) across three main suites:

  • Mathematics: AIME (2022-2026) and OlymMath-Hard.
  • Coding: HumanEval+, MBPP+, and LiveCodeBench v6.
  • General Knowledge: GPQA-Diamond, MMLU-Redux, MMLU-Pro, and ARC-Challenge.

Key Insights and Ideas

  • "Asking for Help Without Words": The most striking insight is that models don't need to "talk" to collaborate. Latent representations capture rich semantic information that a smaller model can leverage more efficiently than raw text.
  • Cross-Family Compatibility: POND works even when the sender and receiver are from entirely different model families (e.g., a GLM sender helping a Qwen receiver). This suggests that independently trained LLMs develop surprisingly compatible internal "languages."
  • Zero-Shot Domain Generalization: Models trained on math and code tasks showed immediate performance boosts on general knowledge tasks without any additional domain-specific fine-tuning. The "bridge" learns a general-purpose way to translate high-level features.
  • Compute Asymmetry: By avoiding large-model decoding, POND saves roughly 90% of the sender-side compute compared to traditional text-based advising.

Key Results

POND consistently turned a 4B-parameter model into a much more capable reasoner:

  • Accuracy Boost: On math and coding tasks, POND (with Qwen3.8-Max) improved the receiver's macro-average accuracy from 63.94% to 68.02%.
  • Knowledge Gains: In general knowledge benchmarks, the average score jumped from 84.77% to 87.18%.
  • Efficiency: Despite the overhead of the bridge, the system often ran faster than the small model alone. This is because the high-quality guidance allowed the receiver to reach the correct answer with fewer "thinking" tokens, reducing total decoding time.
Table 1: Math and code results. Senders are frozen and prefill-only. POND uses one latent transfer; +Review adds periodic prefill updates. Acc.: accuracy/pass@1 (%, higher is better); Token: total decoded receiver tokens, including both thinking and final-response tokens (thousands per 100 runs, lower is better); Speed: end-to-end latency (s/run, lower is better). Bold marks the better of POND and +Review for each metric, including ties. Improve compares the bold value with the receiver alone: accuracy gain in percentage points, or relative reduction in tokens/latency; an upward latency arrow indicates an increase. Avg averages the five tasks.
Table 1. Improve reports accuracy gains (percentage points) or relative token/latency reductions (%) against the receiver; upward latency arrows denote increases. Avg averages all four tasks. Differences and reductions are computed before rounding.

Limitations and Future Work

  • White-Box Requirement: POND requires access to the internal hidden states of the sender model. This makes it difficult to use with "black-box" proprietary APIs (like OpenAI's) that only provide text outputs.
  • Memory Scaling: As the "Periodic Review" adds more latents to the memory, the receiver's attention cost grows. While the authors tested a sparse attention variant to mitigate this, managing very long-form generation remains a challenge.
  • Static Review Policies: Currently, reviews happen at fixed intervals. Future work could involve "adaptive review," where the small model only "asks for help" when its internal uncertainty crosses a certain threshold.

Summary updated

Abstract

Text-based model collaboration requires a large model to generate intermediate responses before a smaller model can use its guidance, adding large-model decoding cost to inference. We introduce POND (Prefill-Only, No-Decode), a method that uses a frozen large sender for prefill and a small receiver for all autoregressive decoding. A trainable bridge maps the sender's hidden states to additional attention keys and values, allowing the receiver to use these representations throughout generation. POND also supports periodic review, in which the sender processes the receiver's partial response and updates its guidance through additional prefill passes while preserving the receiver's existing KV cache. Across reasoning and coding benchmarks, POND improves receiver performance with senders from different model families. In particular, GLM-5.3 provides gains comparable to those of Qwen3.8-Max when paired with a Qwen receiver, supporting the effectiveness of representation transfer between independently trained model families. POND further demonstrates its ability to generalize across domains. When trained on math and coding tasks, it transfers directly to general knowledge tasks without additional domain-specific training. These results demonstrate that large frontier model prefill representations can improve small-model generation without requiring large-model autoregressive decoding.
BibTeX
@misc{wang2026latent,
  author = {Qichao Wang and Tingyu Qu and Pengcheng Wu and Yiran Zhong and Feida Zhu and Zhiqi Shen and Peilin Zhao and Steven Hoi},
  title = {{Latent Bridge: Asking for Help Without Words}},
  year = {2026},
  eprint = {2611.21301},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21301},
}
Figure