Aggregated Learning: A Deep Learning Framework Based on Information-Bottleneck Vector Quantization

Hongyu Guo, Yongyi Mao, Ali Al-Bashabsheh, Richong Zhang

Abstract

Based on the notion of information bottleneck (IB), we formulate a quantization problem called "IB quantization". We show that IB quantization is equivalent to learning based on the IB principle. Under this equivalence, the standard neural network models can be viewed as scalar (single sample) IB quantizers. It is known, from conventional rate-distortion theory, that scalar quantizers are inferior to vector (multi-sample) quantizers. Such a deficiency then inspires us to develop a novel learning framework, AgrLearn, that corresponds to vector IB quantizers for learning with neural networks. Unlike standard networks, AgrLearn simultaneously optimizes against multiple data samples. We experimentally verify that AgrLearn can result in significant improvements when applied to several current deep learning architectures for image recognition and text classification. We also empirically show that AgrLearn can reduce up to 80% of the training samples needed for ResNet training.

Introduction

The Information Bottleneck (IB) principle provides a powerful lens for understanding deep learning, framing the training process as a search for a representation $T$ that compresses input $X$ while preserving information about target $Y$. However, a fundamental theoretical gap exists: standard neural networks process data samples individually, effectively acting as scalar quantizers. In information theory, vector quantizers—which process multiple samples simultaneously—are mathematically proven to be more efficient than scalar ones. This paper introduces Aggregated Learning (AgrLearn), a framework that bridges this gap by treating learning as a vector quantization problem.

The AgrLearn Framework

AgrLearn shifts the learning paradigm from single-sample processing to multi-sample aggregation. The architecture consists of three primary components:

  1. Aggregation (The "Fold"): Multiple input objects (e.g., $n$ images) are concatenated into a single input "fold" $X^n$.
  2. Shared Representation: This fold passes through a shared hidden network $H$, producing a joint bottleneck representation $T^n$.
  3. Parallel Prediction: The representation is fed into $n$ parallel softmax layers, each tasked with predicting the label of one specific object in the fold.
The AgrLearn architecture aggregates $n$ inputs into a single fold, processing them through a shared network to $n$ parallel softmax outputs.

By minimizing the collective cross-entropy loss of all $n$ predictions, the network is forced to learn features that are robust across the entire group, effectively implementing a vector IB quantizer. For inference, a single object is replicated $n$ times to fill the fold, and the model's parallel predictions are averaged.

Experimental Evaluation and Key Insights

The authors integrated AgrLearn into various state-of-the-art architectures, including ResNet and LSTMs, testing them on benchmarks like CIFAR-10, ImageNet, and text classification datasets.

Performance and Data Efficiency

AgrLearn consistently outperformed standard models. For instance, an AgrLearn-enhanced ResNet-18 achieved a 14.4% relative error reduction on CIFAR-10 and a 10.6% reduction on SVHN. Most notably, the framework demonstrated extreme data efficiency: a model trained on just 20% of the CIFAR-10 dataset using AgrLearn matched the performance of a standard ResNet trained on the full 100% of the data.

Implicit Regularization

A key observation during training is the behavior of the loss function. While standard networks often see training loss drop rapidly to zero (leading to potential overfitting), AgrLearn maintains a higher training loss for a longer duration. This is due to the vastly larger combinatorial space of aggregated samples, which acts as a form of implicit regularization, preventing the model from overfitting to individual samples.

Training dynamics of ResNet vs. AgrLearn-ResNet. AgrLearn maintains a higher training loss while achieving a lower test error, indicating superior generalization.

Conclusion

AgrLearn provides a theoretically grounded method to enhance neural network performance by moving from scalar to vector-based learning. While it introduces higher memory requirements during training, its significant gains in accuracy and data efficiency make it a compelling "plug-and-play" framework for modern deep learning tasks. Future work will focus on optimizing the "fold size" and reducing the computational overhead associated with large-scale aggregation.

Summary updated

Abstract

Based on the notion of information bottleneck (IB), we formulate a quantization problem called "IB quantization". We show that IB quantization is equivalent to learning based on the IB principle. Under this equivalence, the standard neural network models can be viewed as scalar (single sample) IB quantizers. It is known, from conventional rate-distortion theory, that scalar quantizers are inferior to vector (multi-sample) quantizers. Such a deficiency then inspires us to develop a novel learning framework, AgrLearn, that corresponds to vector IB quantizers for learning with neural networks. Unlike standard networks, AgrLearn simultaneously optimizes against multiple data samples. We experimentally verify that AgrLearn can result in significant improvements when applied to several current deep learning architectures for image recognition and text classification. We also empirically show that AgrLearn can reduce up to 80% of the training samples needed for ResNet training.
BibTeX
@misc{guo2018aggregated,
  author = {Hongyu Guo and Yongyi Mao and Ali Al-Bashabsheh and Richong Zhang},
  title = {{Aggregated Learning: A Deep Learning Framework Based on Information-Bottleneck Vector Quantization}},
  year = {2018},
  eprint = {1807.10251},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1807.10251},
}
Figure