Practical Semantic Communication for General-Purpose Networks: Compression of High-Dimensional Embeddings Under Quantization

Leonardo Roque, Louis-Adrien Dufrène, Quentin Lampin, Grégoire Lefebvre, Mohamad AssaadarXiv 2026 · arXiv

TL;DR

This paper introduces a practical framework for semantic communication that compresses high-dimensional text embeddings while preserving the information most critical for specific downstream tasks. By "relaxing" the classical Gaussian Information Bottleneck (GIB) model, the authors provide a mathematically grounded way to handle discrete classification labels and noisy digital channels, achieving up to an 85x reduction in bit budget compared to standard task-agnostic compression methods.

What Problem Does This Paper Address?

Traditional wireless communication follows Shannon’s paradigm: transmit every bit reliably, regardless of what those bits actually mean. As we move toward 6G and general-purpose AI networks, this approach is becoming a bottleneck. Semantic communication shifts the goal: transmit only the information necessary to complete a task (like classifying a text's intent).

However, existing semantic communication methods often fall into two traps:

  1. Task-Specific "Black Boxes": Many use end-to-end deep learning (Deep JSCC) that requires retraining the entire system for every new task, making them hard to deploy in general-purpose networks.
  2. Unrealistic Theory: Elegant theoretical models like the Gaussian Information Bottleneck (GIB) assume data and targets are perfectly Gaussian—an assumption that fails for discrete text classification and real-world embeddings.

The authors bridge this gap by creating a modular, theoretically sound framework that works with off-the-shelf foundation models and accounts for the "digital chain" (quantization and channel noise).

Methods

The authors propose a Relaxed GIB framework. Here is the breakdown of their "Digital Chain" approach:

  1. Embedding Generation: They use a pre-trained foundation model (like mpnet-base-nli) to turn raw text into high-dimensional (768-D) embeddings ($X$).
  2. Linear Compression: A compression matrix ($A$) reduces the embedding to a lower-dimensional space ($T$). Unlike standard GIB, they only assume class-conditional Gaussianity. This means that while the overall data distribution can be complex, the data within a specific class is assumed to be Gaussian. This allows the model to handle discrete labels ($Y$).
  3. Optimization via Analytical Gradients: The authors derived closed-form expressions for the gradients of the Information Bottleneck objective. This allows them to optimize the compression matrix $A$ to maximize task-relevant information while minimizing the bit rate.
  4. Quantization & Transmission: The compressed vectors are quantized using K-means to produce a bit sequence ($B$), which is then transmitted over a noisy Binary Symmetric Channel (BSC).
  5. Reconstruction & Inference: The receiver reconstructs the embedding and passes it to a simple linear classifier.
Fig. 1: Schematic of the studied semantic communication system.

Datasets

The framework was evaluated using the Banking77 dataset, accessed via the Massive Text Embedding Benchmark (MTEB).

  • Banking77: A fine-grained intent recognition dataset with 13,073 customer service queries across 77 different categories.
  • Significance: The large number of classes (77) makes this a challenging "stress test" for semantic preservation compared to simple binary classification.

Key Insights and Ideas

  • Breaking the Gaussian Constraint: By moving from "jointly Gaussian" to "class-conditional Gaussian," the authors made GIB practical for classification. They proved that the source marginal can be a Gaussian mixture, which is a much more realistic model for clustered embedding spaces.
  • The Distortion Bound: The authors established a rigorous mathematical bound (using Lipschitz continuity) linking continuous optimization to discrete performance. They proved that if you minimize quantization distortion (via K-means), you are effectively preserving the mutual information optimized in the continuous GIB stage.
  • Information "Pruning": The results suggest that high-dimensional embeddings contain significant redundancy. The optimized compression matrix $A$ acts as a specialized feature extractor, identifying the "semantic core" needed for the task and discarding the rest.

Key Results

  • Massive Efficiency Gains: To achieve ~83% accuracy on Banking77, the proposed method required only 9 bits per embedding. A standard task-agnostic method (Product Quantization) required 768 bits to reach similar performance—an 85x improvement in spectral efficiency.
  • Approaching the Shannon Limit: The method's performance curves closely approach the theoretical Shannon entropy limit of the labels, suggesting it is extracting nearly all available task-relevant information.
  • Robustness to Noise: The system maintained high fidelity even under channel errors (bit flip probability $p_{error} = 0.01$), demonstrating that the learned representations are inherently robust.
  • Dimensionality Reduction: The authors successfully compressed 768-dimensional embeddings down to 32 and 128 dimensions with negligible loss in classification accuracy.

Limitations and Future Work

  • Monte Carlo Sensitivity: The optimization relies on Monte Carlo sampling ($K$). The authors noted that as the codebook size ($L$) increases, the number of samples required to estimate gradients accurately also increases, which can be computationally expensive.
  • Generalization Gap: There is a slight performance drop when moving from training centroids to test embeddings at very high quantization resolutions, suggesting a "generalization gap" that needs further study.
  • Beyond Text: While this paper focused on text embeddings, the framework is theoretically applicable to any modality (images, sensor data) that can be represented as high-dimensional embeddings. Testing this on multimodal data is a logical next step.

Summary updated

Abstract

This paper introduces a practical framework for semantic communication over wireless channels, focusing on the compression of high-dimensional embeddings under quantization and channel errors. Building upon a relaxed Gaussian Information Bottleneck (GIB) model, the approach provides analytical solutions for task-relevant information preservation, even with non-Gaussian targets. The framework relaxes traditional Gaussian assumptions, enabling effective compression for discrete and complex data such as text embeddings. It derives closed-form gradients for optimizing the compression matrix and establishes bounds linking continuous representations to discrete, noisy system performance. Comprehensive experiments on text classification tasks demonstrate that the proposed method outperforms task-agnostic quantization-based compression methods, maintaining high semantic fidelity at low bit budgets per embedding. The approach offers a scalable, interpretable, and resource-efficient solution for deploying semantic communication in general-purpose networks.
BibTeX
@misc{roque2026practical,
  author = {Leonardo Roque and Louis-Adrien Dufrène and Quentin Lampin and Grégoire Lefebvre and Mohamad Assaad},
  title = {{Practical Semantic Communication for General-Purpose Networks: Compression of High-Dimensional Embeddings Under Quantization}},
  year = {2026},
  eprint = {2611.20863},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20863},
}
Figure