Abstract
Based on the notion of information bottleneck (IB), we formulate a quantization problem called "IB quantization". We show that IB quantization is equivalent to learning based on the IB principle. Under this equivalence, the standard neural network models can be viewed as scalar (single sample) IB quantizers. It is known, from conventional rate-distortion theory, that scalar quantizers are inferior to vector (multi-sample) quantizers. Such a deficiency then inspires us to develop a novel learning framework, AgrLearn, that corresponds to vector IB quantizers for learning with neural networks. Unlike standard networks, AgrLearn simultaneously optimizes against multiple data samples. We experimentally verify that AgrLearn can result in significant improvements when applied to several current deep learning architectures for image recognition and text classification. We also empirically show that AgrLearn can reduce up to 80% of the training samples needed for ResNet training.
Introduction
The Information Bottleneck (IB) principle provides a powerful lens for understanding deep learning, framing the training process as a search for a representation $T$ that compresses input $X$ while preserving information about target $Y$. However, a fundamental theoretical gap exists: standard neural networks process data samples individually, effectively acting as scalar quantizers. In information theory, vector quantizers—which process multiple samples simultaneously—are mathematically proven to be more efficient than scalar ones. This paper introduces Aggregated Learning (AgrLearn), a framework that bridges this gap by treating learning as a vector quantization problem.
The AgrLearn Framework
AgrLearn shifts the learning paradigm from single-sample processing to multi-sample aggregation. The architecture consists of three primary components:
- Aggregation (The "Fold"): Multiple input objects (e.g., $n$ images) are concatenated into a single input "fold" $X^n$.
- Shared Representation: This fold passes through a shared hidden network $H$, producing a joint bottleneck representation $T^n$.
- Parallel Prediction: The representation is fed into $n$ parallel softmax layers, each tasked with predicting the label of one specific object in the fold.
By minimizing the collective cross-entropy loss of all $n$ predictions, the network is forced to learn features that are robust across the entire group, effectively implementing a vector IB quantizer. For inference, a single object is replicated $n$ times to fill the fold, and the model's parallel predictions are averaged.
Experimental Evaluation and Key Insights
The authors integrated AgrLearn into various state-of-the-art architectures, including ResNet and LSTMs, testing them on benchmarks like CIFAR-10, ImageNet, and text classification datasets.
Performance and Data Efficiency
AgrLearn consistently outperformed standard models. For instance, an AgrLearn-enhanced ResNet-18 achieved a 14.4% relative error reduction on CIFAR-10 and a 10.6% reduction on SVHN. Most notably, the framework demonstrated extreme data efficiency: a model trained on just 20% of the CIFAR-10 dataset using AgrLearn matched the performance of a standard ResNet trained on the full 100% of the data.
Implicit Regularization
A key observation during training is the behavior of the loss function. While standard networks often see training loss drop rapidly to zero (leading to potential overfitting), AgrLearn maintains a higher training loss for a longer duration. This is due to the vastly larger combinatorial space of aggregated samples, which acts as a form of implicit regularization, preventing the model from overfitting to individual samples.
Conclusion
AgrLearn provides a theoretically grounded method to enhance neural network performance by moving from scalar to vector-based learning. While it introduces higher memory requirements during training, its significant gains in accuracy and data efficiency make it a compelling "plug-and-play" framework for modern deep learning tasks. Future work will focus on optimizing the "fold size" and reducing the computational overhead associated with large-scale aggregation.
Summary updated