Token Communication in the Era of Large Models: An Information Bottleneck-Based Approach

Hao Wei, Wanli Ni, Wen Wang, Wenjun Xu, Dusit Niyato, Ping Zhang

TL;DR

The UniToCom framework redefines wireless communication by treating "tokens"—the building blocks of Large Language Models—as the fundamental unit for both AI processing and transmission. By applying a novel Generative Information Bottleneck (GenIB) principle, the system learns to transmit only the most essential data needed for a Multimodal Large Language Model (MLLM) at the receiver to accurately reconstruct or reason about the original information.

What Problem Does This Paper Address?

Traditional wireless communication focuses on transmitting bits accurately. Recently, Semantic Communication (SemCom) has emerged, focusing on transmitting high-level "meanings" rather than raw data. However, most SemCom systems use continuous, high-dimensional features that are often "black boxes"—they lack structural granularity and don't play well with the token-based architecture of modern AI like GPT-4 or LLaVA.

The authors identify two core challenges:

  1. Efficiency: How do we extract tokens that are small enough for constrained wireless channels but "rich" enough for high-quality AI generation?
  2. Unified Processing: How can a single communication system handle text, images, and audio simultaneously while maintaining the stability of the AI models at the receiving end?

Methods

The authors introduce UniToCom (Unified Token Communication), which consists of several innovative components:

  • Token-Centric Pipeline: Unlike traditional systems that separate source coding (compression) and channel coding (error correction), UniToCom maps input data directly into latent tokens. These tokens are then processed by an adaptive module for wireless transmission.
  • Generative Information Bottleneck (GenIB): This is the "brain" of the compression. Based on the Information Bottleneck principle, GenIB forces the model to find a "minimal sufficient" representation. It compresses the data as much as possible (the "Rate" term) while ensuring the receiver can still generate the original content (the "Distortion" term).
  • $\sigma$-GenIB: A specific refinement to prevent "variance collapse." In generative AI, if the latent space becomes too narrow, the model loses diversity and stability. $\sigma$-GenIB imposes a fixed variance constraint and blends deterministic and stochastic representations to keep the model robust.
  • Receiver-side MLLM: The receiver uses a 1.4B parameter causal Transformer. It treats communication as a "next-token prediction" task. For discrete data (text), it predicts the next word; for continuous data (images/audio), it uses a diffusion head to refine the latent tokens into high-fidelity outputs.

Datasets

The framework was evaluated across three distinct multimodal tasks using standard benchmarks:

  • CLEVR: Used for Visual Question Answering (VQA) to test understanding.
  • MS-COCO: Used for text-to-image generation to test generative capability.
  • LibriSpeech: Used for Automatic Speech Recognition (ASR) to test audio processing.

Key Insights and Ideas

  • Tokens as the Universal Currency: The paper argues that tokens are the natural "interface" between communication and intelligence. By aligning the transmission format with the model's internal language, the system becomes more scalable and generalizable.
  • The "Goldilocks" Zone of Compression: One of the most interesting findings (shown in Fig. 3) is that longer token sequences aren't always better. Excessively long sequences introduce redundant noise that can actually impair model accuracy. There is an optimal "sweet spot" (e.g., a token length of 32 for images) that balances information density with computational efficiency.
  • Stability through $\sigma$-GenIB: By explicitly managing the variance of the latent tokens, the authors solved a common problem where autoregressive models become unstable or "boring" when fed highly compressed data.

Key Results

UniToCom was tested against traditional schemes (like JPEG/AMR-WB combined with LDPC/Turbo codes) and "Vanilla" SemCom:

  • Robustness to Noise: In low Signal-to-Noise Ratio (SNR) regimes—where traditional methods often fail completely—UniToCom maintained high performance, significantly outperforming the baselines.
  • Semantic Fidelity: On the MS-COCO text-to-image task, the GenIB approach achieved much lower FID scores (a measure of image quality/diversity) compared to standard VAE-based tokenization.
  • Efficiency: The system demonstrated that moderate token compression not only saves bandwidth but also accelerates the convergence of the MLLM, making the whole AI pipeline faster.

Limitations and Future Work

  • Computational Overhead: The receiver requires a 1.4B parameter MLLM. While the authors suggest this is feasible for a base station with "sufficient computing power," it might be a hurdle for direct device-to-device communication on low-power hardware.
  • Dynamic Environments: While tested under Rayleigh fading (a standard wireless model), real-world interference and mobility patterns are more complex.
  • Future Directions: The authors suggest scaling this to even more modalities (like video) and further optimizing the MLLM backbone for edge deployment.

Reference: Wei, H., et al. "Token Communication in the Era of Large Models: An Information Bottleneck-Based Approach." arXiv:2507.01728v1, 2025.

Summary updated

Abstract

This letter proposes UniToCom, a unified token communication paradigm that treats tokens as the fundamental units for both processing and wireless transmission. Specifically, to enable efficient token representations, we propose a generative information bottleneck (GenIB) principle, which facilitates the learning of tokens that preserve essential information while supporting reliable generation across multiple modalities. By doing this, GenIB-based tokenization is conducive to improving the communication efficiency and reducing computational complexity. Additionally, we develop $σ$-GenIB to address the challenges of variance collapse in autoregressive modeling, maintaining representational diversity and stability. Moreover, we employ a causal Transformer-based multimodal large language model (MLLM) at the receiver to unify the processing of both discrete and continuous tokens under the next-token prediction paradigm. Simulation results validate the effectiveness and superiority of the proposed UniToCom compared to baselines under dynamic channel conditions. By integrating token processing with MLLMs, UniToCom enables scalable and generalizable communication in favor of multimodal understanding and generation, providing a potential solution for next-generation intelligent communications.
BibTeX
@article{wei2026token,
  author = {Hao Wei and Wanli Ni and Wen Wang and Wenjun Xu and Dusit Niyato and Ping Zhang},
  title = {{Token Communication in the Era of Large Models: An Information Bottleneck-Based Approach}},
  journal = {IEEE Wireless Communications Letters},
  year = {2026},
  doi = {10.1109/lwc.2025.3622628},
}

Loading related content...

Figure