Multi-view Information Bottleneck Without Variational Approximation

Qi Zhang, Shujian Yu, Jingmin Xin, Badong Chen

Abstract By "intelligently" fusing the complementary information across different views, multi-view learning is able to improve the performance of classification tasks. In this work, we extend the information bottleneck principle to a supervised multi-view learning scenario and use the recently proposed matrix-based Rényi’s $\alpha$-order entropy functional to optimize the resulting objective directly, without the necessity of variational approximation or adversarial training. Empirical results in both synthetic and real-world datasets suggest that our method enjoys improved robustness to noise and redundant information in each view, especially given limited training samples.

Introduction

Multi-view learning seeks to enhance model performance by integrating information from diverse sources, such as video and audio streams. The Information Bottleneck (IB) principle provides a robust framework for this by encouraging models to compress input data ($X$) into a representation ($Z$) that retains only the information essential for predicting a target ($Y$). However, a significant challenge lies in calculating Mutual Information (MI) in high-dimensional spaces. Conventional deep learning methods often rely on variational approximations (e.g., VIB) to estimate MI lower bounds. These approximations can be "loose" and inaccurate, particularly with limited training data, resulting in sub-optimal performance and sensitivity to noise.

Methodology

The proposed Multi-view matrix-based Entropy Information Bottleneck (MEIB) addresses these challenges by bypassing probability distribution estimation. Instead, it utilizes a matrix-based Rényi’s $\alpha$-order entropy functional to optimize the IB objective directly.

The MEIB architecture featuring multiple encoders, a fusion network, and a classifier within an information bottleneck framework.

MEIB calculates entropy directly from the eigenspectrum of a normalized Gram matrix, representing sample similarities within a mini-batch. This approach is fully differentiable, allowing for seamless integration into standard deep learning pipelines and optimization via standard algorithms like Adam or SGD. As shown in the architecture diagram, the model consists of $k$ separate encoders—one for each view—that extract latent features. These features are then integrated by a fusion network before being processed by a classifier. The IB objective is applied to each view independently to ensure individual compression and robustness.

Experimental Setup

The authors evaluated MEIB against several state-of-the-art methods, including Deep Canonical Correlation Analysis (DCCA), its autoencoder variant (DCCAE), and standard Deep IB. Evaluations were conducted on:

  • Synthetic Data: Designed to assess robustness against noise and redundant feature dimensions.
  • Multi-view MNIST: Comprising a rotated image view and an image view with background noise.
  • BBC News: A document dataset categorized into five topics, represented by two distinct feature sets.
  • Wisconsin X-Ray MicroBeam (XRMB): A speech recognition dataset containing acoustic and articulatory data.

Results and Discussion

MEIB demonstrates significant advantages, particularly in scenarios with limited data. Unlike traditional IB methods that must learn underlying distributions, MEIB's direct calculation of MI makes it more effective when sample sizes are small. Furthermore, MEIB provides a deterministic compression method, contrasting with the stochastic nature of Variational Information Bottleneck (VIB).

Classification error comparison across MNIST, BBC, and XRMB datasets.

Key results summarized in the table above include:

  • Noise Robustness: Synthetic tests showed that MEIB maintains lower and more stable classification errors as noise levels increase compared to Linear CCA and standard DNNs.
  • Superior Performance: On the multi-view MNIST task, MEIB achieved a classification error of 0.173, outperforming both Deep IB (0.194) and standard DNNs (0.195).
  • Consistency: MEIB consistently yielded the lowest average classification error across the BBC and XRMB datasets.
  • Redundancy Identification: Weight analysis revealed that MEIB effectively ignores useless features by assigning near-zero weights to redundant dimensions.

Conclusion

While MEIB shows great promise, its computational complexity may scale with mini-batch size due to eigenspectrum calculations. Future research could explore "synergistic information"—the unique insights gained only through the combination of multiple views—which is not explicitly captured by the current additive IB regularization. Additionally, managing the increased number of hyper-parameters associated with multiple views remains a challenge for model tuning.

Summary updated

Abstract

By "intelligently" fusing the complementary information across different views, multi-view learning is able to improve the performance of classification tasks. In this work, we extend the information bottleneck principle to a supervised multi-view learning scenario and use the recently proposed matrix-based R{é}nyi's $α$-order entropy functional to optimize the resulting objective directly, without the necessity of variational approximation or adversarial training. Empirical results in both synthetic and real-world datasets suggest that our method enjoys improved robustness to noise and redundant information in each view, especially given limited training samples. Code is available at~\url{https://github.com/archy666/MEIB}.
BibTeX
@inproceedings{zhang2022multi,
  author = {Qi Zhang and Shujian Yu and Jingmin Xin and Badong Chen},
  title = {{Multi-view Information Bottleneck Without Variational Approximation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  doi = {10.1109/icassp43922.2022.9747614},
}
Figure