Structured IB: Improving Information Bottleneck with Structured Feature Learning

Hanzhe Yang, Youlong Wu, Dingzhu Wen, Yong Zhou, Yuanming ShiAAAI 2025 · DOI

TL;DR

This paper introduces "Structured IB" (SIB), a novel framework that enhances the Information Bottleneck (IB) principle by using multiple auxiliary encoders alongside a main encoder. SIB aims to capture more comprehensive information for better representation learning, leading to improved accuracy and efficiency in deep learning models.

What Problem Does This Paper Address?

The Information Bottleneck (IB) principle is a powerful tool for learning representations that are both informative about a target variable (Y) and compressed with respect to the input (X). This compression is key to improving generalization, robustness, and interpretability in deep neural networks. While the IB Lagrangian method is a popular way to implement IB, its optimization can be challenging and highly sensitive to the design of the feature extractor. Existing methods often rely on complex variational bounds or neural estimators, which can be error-prone and may not perfectly capture the desired information trade-off. This paper addresses the limitation of traditional IB methods that use a single feature extractor, proposing a structured approach to capture richer information.

Methods

The core innovation of this paper is the Structured Information Bottleneck (SIB) framework. Instead of a single encoder, SIB employs a main encoder and multiple auxiliary encoders.

  • Main Encoder: This encoder is trained using the standard IB Lagrangian objective to extract the primary representation.
  • Auxiliary Encoders: These encoders are trained sequentially to capture complementary information that the main encoder might miss. Crucially, they are trained to be informative about the target (Y) but also independent of the features already extracted by the main and previous auxiliary encoders. This independence is enforced by minimizing the mutual information between the new auxiliary feature and the concatenation of previously extracted features.
  • Feature Aggregation: The features from the main and auxiliary encoders are combined using a weighted sum to form the final aggregated feature. The weights are learned during training.
  • Training Process: SIB uses a three-stage training process:
  • Train the main encoder and decoder.
  • Train auxiliary encoders sequentially, enforcing independence.
  • Optimize the weights for feature aggregation.

This structured approach aims to create a more robust and comprehensive representation than a single encoder can achieve.

Datasets

The paper evaluates SIB on two standard benchmark datasets:

  • MNIST: A dataset of handwritten digits.
  • CIFAR-10: A dataset of small color images in 10 classes.

No new datasets were introduced.

Key Insights and Ideas

  • Structured Features are Better: The core idea is that breaking down feature extraction into a main component and complementary auxiliary components can lead to richer representations than a single, monolithic encoder.
  • Independence is Key for Auxiliary Encoders: Training auxiliary encoders to be informative about the target but independent of existing features is crucial for avoiding redundant information and maximizing the benefit of each encoder.
  • Weights Naturally Learn Importance: The learned weights for aggregating features show that the main encoder often receives the largest weight, but auxiliary encoders contribute significantly, especially in more complex tasks.
  • SIB Improves the IB Trade-off: SIB consistently achieves higher mutual information between the compressed representation and the target (I(Z, Y)) for a given level of compression (I(X, Z)) compared to standard IB methods.
  • Parameter Efficiency: SIB can achieve superior performance with fewer parameters than traditional methods, suggesting better utilization of model capacity.

Key Results

  • Improved Accuracy and I(Z, Y): SIB consistently outperforms standard IB Lagrangian methods (VIB, sqVIB, NIB) in terms of prediction accuracy and the mutual information between the learned representation and the target variable (I(Z, Y)) across both MNIST and CIFAR-10 datasets.
  • Comparable I(X, Z): While I(Z, Y) increases, the compression level I(X, Z) either decreases slightly or remains comparable, indicating a better trade-off.
  • Parameter Efficiency: SIB achieves these gains with a reduced number of model parameters, especially when compared to traditional methods. For instance, on MNIST, SIB shows improvements even when the number of parameters is reduced.
  • Weight Behavior: The main encoder consistently receives the largest weight, and auxiliary encoder weights tend to be smaller for MNIST than CIFAR-10, suggesting a greater reliance on auxiliary features for more challenging tasks.

Limitations and Future Work

  • Linear Aggregation: The paper acknowledges that the linear aggregation function for combining features might be a simplification. Future work could explore more sophisticated combination strategies to better capture complex feature space interactions.
  • Theoretical Advancements: While Theorem 1 provides a theoretical basis, further theoretical work is suggested to relax assumptions (like Gaussianity) and explore how to better assist compression.
  • Broader Applicability: The structured approach is theoretically applicable to other information-theoretic methods beyond IB Lagrangian, and future research could explore these extensions.
  • Computational Challenges: The paper notes that some neural estimator-based methods (MINE, KNIFE) encountered convergence issues, highlighting the stability benefits of the proposed SIB framework.

Summary updated

BibTeX
@inproceedings{yang2025structured,
  author = {Hanzhe Yang and Youlong Wu and Dingzhu Wen and Yong Zhou and Yuanming Shi},
  title = {{Structured IB: Improving Information Bottleneck with Structured Feature Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i20.35499},
}
Figure