Exploring Information Processing in Large Language Models: Insights from Information Bottleneck Theory

Zhou Yang, Zhengyu Qi, Zhaochun Ren, Zhikai Jia, Haizhou Sun, Xiaofei Zhu, Xiangwen LiaoarXiv 2025 · arXiv · OpenReview

Abstract

Large Language Models (LLMs) have demonstrated remarkable performance across a wide range of tasks by understanding input information and predicting corresponding outputs. However, the internal mechanisms by which LLMs comprehend input and make effective predictions remain poorly understood. In this paper, we explore the working mechanism of LLMs in information processing from the perspective of Information Bottleneck Theory. We propose a non-training construction strategy to define a task space and identify the following key findings: (1) LLMs compress input information into specific task spaces (e.g., sentiment space, topic space) to facilitate task understanding; (2) they then extract and utilize relevant information from the task space at critical moments to generate accurate predictions. Based on these insights, we introduce two novel approaches: an Information Compression-based Context Learning (IC-ICL) and a Task-Space-guided Fine-Tuning (TS-FT). IC-ICL enhances reasoning performance and inference efficiency by compressing retrieved example information into the task space. TS-FT employs a space-guided loss to fine-tune LLMs, encouraging the learning of more effective compression and selection mechanisms. Experiments across multiple datasets validate the effectiveness of task space construction. Additionally, IC-ICL not only improves performance but also accelerates inference speed by over 40%, while TS-FT achieves superior results with a minimal strategy adjustment.

Introduction

Despite the meteoric rise of Large Language Models (LLMs), the internal mechanics of how they transform a prompt into a coherent response remain largely a "black box." While we observe their output, the "how" of internal information flow is elusive. This research applies Information Bottleneck (IB) Theory—a framework describing how systems discard noise while retaining relevant information—to investigate how LLMs comprehend tasks.

The study identifies a specific failure point: the decompression phase. While models excel at identifying the "gist" of a task (compression), they often struggle to extract and utilize that specific information to generate final outputs (decompression).

Methodology: Mapping the Task Space

To investigate these internal mechanics, the researchers developed several novel techniques to map and improve information flow:

Task-Space Detection

A non-gradient strategy was used to define a "task space" (e.g., an "emotion space" for sentiment analysis). By applying Principal Component Analysis (PCA) on hidden state differences between positive and negative prompts, a coordinate system for specific concepts within the model's internal representations was created.

2D visualization of the constructed emotion task space using t-SNE, showing distinct clustering of emotional categories.

Mutual Information (MI) Estimation

Using K-Nearest Neighbors (KNN)-based methods, the team measured shared information between different model layers and the target task. This allowed for direct observation of the model compressing data in middle layers.

Novel Enhancement Techniques

  1. Information Compression-based Context Learning (IC-ICL): Instead of providing raw text examples in a prompt, this method retrieves relevant examples, maps them into the task space, and injects these compressed representations directly into the hidden layers.
  2. Task-Space-guided Fine-Tuning (TS-FT): This training approach uses a specialized Mean Squared Error (MSE) loss function to "coach" the model's hidden layers toward higher-quality directions within the task space, improving decompression capabilities.

Key Insights and Results

The Compression Sweet Spot

The study found that LLMs do not compress information immediately. In models like Llama-3.1-8B, the primary work of discarding syntax and grammar in favor of task-specific semantics occurs between layers 12 and 28. This is where the model effectively "squishes" input information into task spaces.

Mutual Information vs. Layer, illustrating the dynamics of information compression and the relationship between input (x), hidden states (z), and output (y).

Performance and Efficiency

The proposed methods yielded significant improvements on the Empathetic Dialogues dataset, which focuses on multi-dimensional emotion recognition:

  • IC-ICL Performance: Accuracy reached 43.69%, significantly outperforming standard In-Context Learning (29.55%) and zero-shot baselines (35.35%).
  • Inference Speedup: By using compressed representations rather than long text examples, IC-ICL accelerated inference speed by over 40% compared to standard ICL (7:37 vs 14:17 in test environments).
  • TS-FT Superiority: The fine-tuning method achieved the highest overall accuracy at 55.43%, demonstrating that guiding a model's internal spatial logic is more effective than standard fine-tuning (54.48%).

Conclusion

This work reveals that LLMs process information through a distinct compression-decompression cycle. While they are highly effective at compressing input into task-specific spaces, the bottleneck often lies in decompression—the ability to extract that information for prediction. Future work will focus on automating task space discovery and exploring these dynamics in larger models (70B+) and more abstract reasoning tasks.

Summary updated

Abstract

Large Language Models (LLMs) have demonstrated remarkable performance across a wide range of tasks by understanding input information and predicting corresponding outputs. However, the internal mechanisms by which LLMs comprehend input and make effective predictions remain poorly understood. In this paper, we explore the working mechanism of LLMs in information processing from the perspective of Information Bottleneck Theory. We propose a non-training construction strategy to define a task space and identify the following key findings: (1) LLMs compress input information into specific task spaces (e.g., sentiment space, topic space) to facilitate task understanding; (2) they then extract and utilize relevant information from the task space at critical moments to generate accurate predictions. Based on these insights, we introduce two novel approaches: an Information Compression-based Context Learning (IC-ICL) and a Task-Space-guided Fine-Tuning (TS-FT). IC-ICL enhances reasoning performance and inference efficiency by compressing retrieved example information into the task space. TS-FT employs a space-guided loss to fine-tune LLMs, encouraging the learning of more effective compression and selection mechanisms. Experiments across multiple datasets validate the effectiveness of task space construction. Additionally, IC-ICL not only improves performance but also accelerates inference speed by over 40\%, while TS-FT achieves superior results with a minimal strategy adjustment.
BibTeX
@misc{yang2025exploring,
  author = {Zhou Yang and Zhengyu Qi and Zhaochun Ren and Zhikai Jia and Haizhou Sun and Xiaofei Zhu and Xiangwen Liao},
  title = {{Exploring Information Processing in Large Language Models: Insights from Information Bottleneck Theory}},
  year = {2025},
  eprint = {2501.00999},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.00999},
}
Figure