TL;DR
POND (Prefill-Only, No-Decode) is a new framework that allows a large "sender" model to guide a small "receiver" model by sharing internal latent representations instead of generated text. By skipping the slow, token-by-token decoding of the large model and using only its fast parallel "prefill" pass, POND achieves significant accuracy gains in reasoning and coding while drastically reducing inference latency.
What Problem Does This Paper Address?
In the current landscape of AI, we often want the "brains" of a frontier-scale model (like GPT-4 or Qwen-Max) to assist a smaller, more efficient model running on edge hardware. Traditionally, this collaboration happens via natural language: the big model writes a hint or a chain-of-thought, and the small model reads it.
The bottleneck is latency. Large models are notoriously slow at autoregressive decoding—generating text one token at a time. However, they are incredibly efficient at "prefilling," where they process all input tokens in parallel. The authors ask a compelling question: Can we transfer the intelligence of a large model's prefill representations directly to a small model, skipping the text generation step entirely?
Methods
The authors introduce POND, which decouples the large model's understanding from its writing.
- The Latent Bridge: A frozen large "sender" model prefills the input prompt. A lightweight, trainable "bridge" (an MLP) compresses the sender's final-layer hidden states into a compact latent message.
- KV Adapters: This message is sent to the small "receiver" model. Layer-specific adapters project these latents into additional Attention Keys and Values (KV). Essentially, the small model "sees" the large model's thoughts as extra context in its own attention mechanism.
- Gated Joint Attention: The receiver uses a learned gate to balance its own internal representations with the guidance provided by the sender.
- Periodic Review: For long responses, the initial guidance can "go stale." POND allows the sender to periodically re-read the receiver's partial output. Because of "prefix determinism," the sender only needs to prefill the new tokens, updating the latent memory without recomputing everything.
Datasets
The researchers evaluated POND using a Qwen3.5-4B receiver paired with various frontier senders (Qwen3.8-Max, GLM-5.3, DeepSeek-V4-Flash) across three main suites:
- Mathematics: AIME (2022-2026) and OlymMath-Hard.
- Coding: HumanEval+, MBPP+, and LiveCodeBench v6.
- General Knowledge: GPQA-Diamond, MMLU-Redux, MMLU-Pro, and ARC-Challenge.
Key Insights and Ideas
- "Asking for Help Without Words": The most striking insight is that models don't need to "talk" to collaborate. Latent representations capture rich semantic information that a smaller model can leverage more efficiently than raw text.
- Cross-Family Compatibility: POND works even when the sender and receiver are from entirely different model families (e.g., a GLM sender helping a Qwen receiver). This suggests that independently trained LLMs develop surprisingly compatible internal "languages."
- Zero-Shot Domain Generalization: Models trained on math and code tasks showed immediate performance boosts on general knowledge tasks without any additional domain-specific fine-tuning. The "bridge" learns a general-purpose way to translate high-level features.
- Compute Asymmetry: By avoiding large-model decoding, POND saves roughly 90% of the sender-side compute compared to traditional text-based advising.
Key Results
POND consistently turned a 4B-parameter model into a much more capable reasoner:
- Accuracy Boost: On math and coding tasks, POND (with Qwen3.8-Max) improved the receiver's macro-average accuracy from 63.94% to 68.02%.
- Knowledge Gains: In general knowledge benchmarks, the average score jumped from 84.77% to 87.18%.
- Efficiency: Despite the overhead of the bridge, the system often ran faster than the small model alone. This is because the high-quality guidance allowed the receiver to reach the correct answer with fewer "thinking" tokens, reducing total decoding time.
Limitations and Future Work
- White-Box Requirement: POND requires access to the internal hidden states of the sender model. This makes it difficult to use with "black-box" proprietary APIs (like OpenAI's) that only provide text outputs.
- Memory Scaling: As the "Periodic Review" adds more latents to the memory, the receiver's attention cost grows. While the authors tested a sparse attention variant to mitigate this, managing very long-form generation remains a challenge.
- Static Review Policies: Currently, reviews happen at fixed intervals. Future work could involve "adaptive review," where the small model only "asks for help" when its internal uncertainty crosses a certain threshold.
Summary updated