When humans communicate, we compress our complex, high-dimensional thoughts into a narrow stream of discrete symbols: words. This is an evolutionary necessity because we lack direct neural connections. However, when artificial neural networks communicate—whether they are different modules in a pipeline or independent agents in a multi-agent system—forcing them to use discrete tokens (like natural language text or categorical tags) introduces an artificial and highly inefficient bottleneck.
Instead of decoding internal representations into text only for the receiving model to re-encode them, a growing body of research explores latent state communication. By directly exchanging high-dimensional activations, hidden states, or Key-Value (KV) caches, models can bypass the lossy discretization process entirely.
The Information Bottleneck of Discretization
When a language model generates text, it must commit to a single linear sequence of tokens. This process discards the model's internal uncertainty, soft probabilities, and alternative hypotheses. As argued by Zhuoyun Du et al. in Enabling Agents to Communicate Entirely in Latent Space, communicating via high-dimensional latent states allows agents to transmit a superposition of multiple reasoning paths simultaneously. This preservation of parallel hypotheses gives the receiving agent far more flexibility to explore complex environments.
Similarly, in question-answering and classification tasks, Zixian Huang et al. demonstrated in Clues Before Answers: Generation-Enhanced Multiple-Choice QA that passing high-dimensional hidden states directly to a classifier preserves subtle nuances and certainty levels that are completely lost when information is compressed into a lossy text format. When models are forced to communicate via discrete tokens, they suffer from an information bottleneck that strips away the rich distributional context captured by neural layers, a limitation highlighted by Nanyun Peng et al. in Improving Named Entity Recognition for Chinese Social Media with Word Segmentation Representation Learning.
Eliminating Error Propagation in Multi-Task Pipelines
The benefits of latent communication extend to traditional natural language processing and computer vision pipelines. In a classic cascaded system (such as a word segmenter feeding a named entity recognizer, or a syntax parser feeding a sentiment analyzer), a mistake made by the first component permanently corrupts the input to the second. This is known as error propagation.
By replacing discrete output tags with continuous hidden-layer vectors, the downstream model can access the full representation space of the upstream task. For instance, Yue Zhang et al. showed in Implicit Syntactic Features for Target-dependent Sentiment Analysis that extracting hidden layer vectors from a pre-trained model preserves rich structural information and mitigates the risk of error propagation. The downstream model can resolve ambiguities that hard classification tags would otherwise obscure, a mechanism leveraged by Meishan Zhang et al. in both Enhancing Opinion Role Labeling with Semantic-Aware Word Representations from Semantic Role Labeling and Transition-based Neural RST Parsing with Implicit Syntax Features.
This principle also holds true across modalities. In TextOCR: Towards Large-Scale End-to-End Reasoning for Arbitrary-Shaped Scene Text, the authors found that passing high-dimensional hidden states from an OCR decoder, rather than discrete character strings, preserves vital visual and contextual information for downstream visual reasoning.
Bypassing the Autoregressive Bottleneck
Beyond preserving information, text-based communication between large language models is computationally extravagant. Autoregressive token generation requires generating tokens one by one, which is highly latency-bound and expensive.
Latent state communication fundamentally changes this dynamic by allowing models to swap internal vector representations directly. As shown by Vignav Ramesh et al. in Communicating Activations Between Language Model Agents, exchanging high-dimensional activations improves reasoning performance while eliminating the computational overhead of text generation.
A particularly powerful mechanism for this is the direct transfer and fusion of Key-Value (KV) caches. Instead of generating text and forcing the receiver to re-compute attention keys and values, frameworks like Cache-to-Cache (C2C), proposed by Tianyu Fu et al. in Cache-to-Cache: Direct Semantic Communication Between Large Language Models, project and fuse internal KV caches directly, achieving substantial speedups and higher accuracy. This approach transforms multi-agent collaboration from a slow, sequential debate into a high-bandwidth, parallelized exchange of underlying reasoning vectors, as explored by Yujia Zheng et al. in Thought Communication in Multiagent Collaboration and Haibo Jin et al. in Agent Primitives: Reuseable Latent Building Blocks for Multi-Agent Systems.
Making Latent Communication Robust and Context-Agnostic
If latent communication is so powerful, why hasn't it completely replaced text? The primary challenge is that a model's hidden states are typically highly dependent on its specific local prompt context and internal architecture. If Agent A and Agent B have different prompts, Agent A's raw hidden states might be uninterpretable to Agent B.
To solve this, researchers have developed methods to isolate the universal logical components of latent states. In Augmenting Multi-Agent Communication with State Delta Trajectory, the authors introduced State Delta Encoding (SDE), which transmits the changes (deltas) in a model's internal hidden states rather than the raw states themselves. Because these deltas represent the universal logical steps taken during reasoning, the receiving agent can seamlessly integrate the sender's cognitive trajectory regardless of differences in their individual local instructions.
[Agent A Context + Prompt] ──> Compute State Changes (Deltas) ──> [Universal Logical Step]
│
Latent Transfer
│
▼
[Agent B Context + Prompt] <─── Integrate State Deltas <──────────────────┘
Furthermore, when dealing with heterogeneous agents or task-specific adaptation, lightweight modulators can be trained to refine these high-dimensional hidden states directly, capturing nuanced patterns that are completely invisible when looking only at final token probabilities, as demonstrated by Jiahao Li et al. in Feature-Adaptive and Data-Scalable In-Context Learning.
This shifts the paradigm of machine-to-machine interaction away from human-centric interfaces toward a native, semantic internet of autonomous models, whether applied to virtual multi-agent systems or embodied robotic controllers (Yuntao Wang et al. in Internet of Agents: Fundamentals, Applications, and Challenges; Zhiyu Huang et al. in TIC-VLA: A Think-in-Control Vision-Language-Action Model for Robot Navigation in Dynamic Environments).
Key Foundational Work
Improving Named Entity Recognition for Chinese Social Media with Word Segmentation Representation Learning (2016) — contributed 2 concept elements to this direction.
Implicit Syntactic Features for Target-dependent Sentiment Analysis (2017) — contributed 1 concept elements to this direction.