Abstract
Based on the notion of information bottleneck (IB), we formulate a quantization problem called "IB quantization". We show that IB quantization is equivalent to learning based on the IB principle. Under this equivalence, the standard neural network models can be viewed as scalar (single sample) IB quantizers. It is known, from conventional rate-distortion theory, that scalar quantizers are inferior to vector (multi-sample) quantizers. Such a deficiency then inspires us to develop a novel learning framework, AgrLearn, that corresponds to vector IB quantizers for learning with neural networks. Unlike standard networks, AgrLearn simultaneously optimizes against multiple data samples. We experimentally verify that AgrLearn can result in significant improvements when applied to several current deep learning architectures for image recognition and text classification. We also empirically show that AgrLearn can reduce up to 80% of the training samples needed for ResNet training.
Introduction
In deep learning, the Information Bottleneck (IB) principle suggests that an effective model should compress the input $X$ into a representation $T$ that discards irrelevant noise while retaining maximum information about the label $Y$.
Standard neural networks typically process samples individually. In information theory, this is analogous to "scalar quantization." However, rate-distortion theory establishes that scalar quantizers are mathematically inferior to "vector quantizers"—systems that process multiple signals simultaneously. This paper identifies a theoretical gap: if current neural networks function as scalar quantizers, they are likely underperforming. The authors bridge this gap by introducing Aggregated Learning (AgrLearn), a framework that enables neural networks to operate as vector quantizers.
Methods
AgrLearn is designed as a flexible framework that can be integrated into existing deep learning architectures with minimal modifications.
Theoretical Equivalence
The paper establishes that the objective of IB learning is mathematically equivalent to IB quantization. This theoretical foundation justifies the application of quantization theory to enhance neural network learning by moving from scalar to vector representations.
The "Fold-n" Architecture
Instead of processing a single image, AgrLearn utilizes a "Fold-n" approach. It takes $n$ random images and concatenates them into a single "amalgamated object." The network processes this combined input through its standard layers to a final hidden layer, which acts as the bottleneck. This bottleneck is then passed to $n$ parallel softmax layers, each predicting the label for one of the $n$ original objects.
During training, the model minimizes the joint cross-entropy loss of all $n$ labels. For inference, a single test image is replicated $n$ times, and the average of the $n$ resulting distributions provides the final prediction.
Experimental Evaluation
The authors evaluated AgrLearn across several domains and architectures:
- Computer Vision: Cifar10, Cifar100, SVHN, and ImageNet-2012.
- Natural Language Processing (NLP): Movie Review and Subjectivity benchmarks.
- Architectures: ResNet-18, WideResNet-22-10, and standard CNN/LSTM models for text.
Key Results
- Accuracy Improvements: AgrLearn consistently outperformed standard models, achieving a 14.4% relative error reduction on Cifar10 and an 11.3% reduction in Top-5 error on ImageNet.
- NLP Gains: Accuracy on the Movie Review dataset improved from 76.1% to 79.3% using an AgrLearn-CNN.
- Data Efficiency: AgrLearn is remarkably robust to data scarcity; a model trained on only 20% of Cifar10 using AgrLearn can outperform a standard model trained on the full dataset.
Insights and Discussion
- Vectorization Benefits: By representing $n$ objects simultaneously, the network is forced to identify more efficient, joint features that single-sample models might overlook. Visualization of feature maps confirms that the model learns contextual, joint representations.
- Implicit Regularization: AgrLearn serves as a powerful regularizer. The vast space of amalgamated examples prevents the model from memorizing individual samples, thereby encouraging better generalization.
- Scaling: AgrLearn benefits from wider models, as increased input complexity requires more capacity to extract joint features effectively.
Limitations
- Memory Overhead: Processing $n$ objects simultaneously increases memory consumption roughly $n$-fold.
- The "Critical n": While larger folds theoretically improve compression, there is likely a point where empirical noise in finite training data causes performance to degrade.
- Optimization Proxies: The current implementation uses cross-entropy as a proxy for the IB objective. Future work may explore direct optimization of the IB objective within this framework.
Summary updated