Representation Learning with Conditional Information Flow Maximization

Dou Hu, Lingwei Wei, Wei Zhou, Songlin Hu

Abstract

This paper proposes an information-theoretic representation learning framework, named conditional information flow maximization (CIFM), to extract noise-invariant sufficient representations for input data and target tasks. It promotes learned representations with good feature uniformity and sufficient predictive ability, enhancing the generalization of pre-trained language models (PLMs). Firstly, an information flow maximization principle is proposed to learn more sufficient representations by simultaneously maximizing both input-representation and representation-label mutual information. Unlike the information bottleneck, we handle input-representation information in an opposite way to avoid over-compression. Additionally, to mitigate the negative effect of potential redundant features, we design a conditional information minimization principle to eliminate negative redundant features while preserving noise-invariant features. Experiments on 13 language understanding benchmarks demonstrate that CIFM effectively improves PLM performance for classification and regression, yielding representations that are more sufficient, robust, and transferable.

Introduction

In deep learning, representation learning transforms raw data into a compact mathematical format suitable for downstream tasks. Historically, two paradigms have dominated this space: InfoMax, which seeks to retain as much input information as possible, and the Information Bottleneck (IB), which compresses the input to retain only task-essential information.

However, both approaches face a "sufficiency-compression" conflict. InfoMax often retains harmful noise and irrelevant details, leading to biased models. Conversely, IB can over-compress the latent space, discarding useful context necessary for generalization to unseen scenarios. This paper introduces CIFM to resolve this conflict by extracting "noise-invariant sufficient representations"—features rich enough for utility but stable enough to ignore random fluctuations.

Methodology

The CIFM framework operates on two core principles designed to balance information flow and noise filtering:

  1. Information Flow Maximization (IFM): Unlike IB, which minimizes the relationship between input ($X$) and representation ($Z$), IFM maximizes it. By simultaneously maximizing the mutual information between $(X, Z)$ and $(Z, Y)$, the model ensures it retains the "necessary ingredients" for the target task.
  2. Conditional Information Minimization (CIM): This acts as a filter to prevent the model from memorizing noise. It targets "negative redundant features"—nuisances that create spurious correlations—using Fisher Information and Adversarial Training. By perturbing model weights during training, CIFM identifies and ignores "brittle" noise while preserving stable, noise-invariant features.
Conceptual comparison of InfoMax, Information Bottleneck (IB), and the proposed CIFM framework.

This "anti-bottleneck" approach suggests that maximizing input-representation information is beneficial if paired with a robust mechanism to filter negative redundancy. CIFM distinguishes between harmful redundancy (noise causing bias) and non-negative redundancy (stable features that aid robustness).

Experimental Setup

The authors evaluated CIFM across 13 natural language understanding benchmarks using BERT and RoBERTa backbones:

  • 10 Classification Tasks: Including EmojiEval, HatEval (hate speech), IronyEval, and emotion datasets like GoEmotions and MELD.
  • 3 Regression Tasks: STS-B (semantic similarity), CLAIRE, and EmoBank.

Results and Discussion

CIFM consistently outperformed standard Cross-Entropy (CE) and IB methods. On the RoBERTa backbone, it achieved an average performance boost of +3.8% for classification and +1.9% for regression.

Performance comparison of CIFM against standard Cross-Entropy (CE) across various NLU benchmarks.

Key Findings:

  • High-Gain Tasks: Significant improvements were noted on complex datasets like HatEval (+14.1% macro-F1) and IronyEval (+5.0%), where subtle semantic cues are critical.
  • Robustness and Generalization: In Out-of-Distribution tests, CIFM improved average scores by +7.9%, indicating it learns underlying concepts rather than surface patterns. The focus on noise-invariant features naturally increased resistance to adversarial attacks.
  • Data Efficiency: CIFM maintained superior performance even when trained on only 20% of available data, demonstrating its ability to extract high-quality representations from limited signal.

Conclusion

Conditional Information Flow Maximization (CIFM) provides a robust information-theoretic framework for representation learning. By balancing information sufficiency with targeted noise minimization, it enhances the performance and transferability of pre-trained language models. While the current study focuses on classification and regression, future work may explore its application in generative tasks and further investigate the theoretical dynamics of information flow. Despite the added computational overhead of adversarial training, the gains in robustness and efficiency make CIFM a compelling alternative to traditional bottleneck methods.

Summary updated

Abstract

This paper proposes an information-theoretic representation learning framework, named conditional information flow maximization, to extract noise-invariant sufficient representations for the input data and target task. It promotes the learned representations have good feature uniformity and sufficient predictive ability, which can enhance the generalization of pre-trained language models (PLMs) for the target task. Firstly, an information flow maximization principle is proposed to learn more sufficient representations for the input and target by simultaneously maximizing both input-representation and representation-label mutual information. Unlike the information bottleneck, we handle the input-representation information in an opposite way to avoid the over-compression issue of latent representations. Besides, to mitigate the negative effect of potential redundant features from the input, we design a conditional information minimization principle to eliminate negative redundant features while preserve noise-invariant features. Experiments on 13 language understanding benchmarks demonstrate that our method effectively improves the performance of PLMs for classification and regression. Extensive experiments show that the learned representations are more sufficient, robust and transferable.
BibTeX
@inproceedings{hu2024representation,
  author = {Dou Hu and Lingwei Wei and Wei Zhou and Songlin Hu},
  title = {{Representation Learning with Conditional Information Flow Maximization}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=Dqp8mUUAuv},
}
Figure