Information Bottleneck and Aggregated Learning

Masoumeh Soflaei, Richong Zhang, Hongyu Guo, Ali Al-Bashabsheh, Yongyi Mao

Abstract

We consider the problem of learning a neural network classifier. Under the information bottleneck (IB) principle, we associate with this classification problem a representation learning problem, which we call “IB learning”. We show that IB learning is, in fact, equivalent to a special class of the quantization problem. The classical results in rate-distortion theory then suggest that IB learning can benefit from a “vector quantization” approach, namely, simultaneously learning the representations of multiple input objects. Such an approach assisted with some variational techniques, result in a novel learning framework, “Aggregated Learning”, for classification with neural network models. In this framework, several objects are jointly classified by a single neural network. The effectiveness of this framework is verified through extensive experiments on standard image recognition and text classification tasks.
BibTeX
@article{soflaei2023information,
  author = {Masoumeh Soflaei and Richong Zhang and Hongyu Guo and Ali Al-Bashabsheh and Yongyi Mao},
  title = {{Information Bottleneck and Aggregated Learning}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2023},
  doi = {10.1109/tpami.2023.3302150},
}

Loading related content...

Figure