Abstract
Knowledge distillation (KD) aims to transfer information from a complex teacher model to a simpler student model. Traditional methods often rely on Euclidean metrics for representation matching, which fail to account for the non-flat geometry of neural network activations. We propose Extended Bregman PCA, a dimensionality reduction technique that utilizes Bregman divergences to respect the intrinsic geometry of activation functions. By integrating this into a distillation framework called Bregman Representation Learning (BRL), we enable students to learn teacher-derived coordinate systems. Our results show that BRL significantly improves performance and sample efficiency across CIFAR and ImageNet benchmarks, particularly when training data is limited.
Introduction
In deep learning, intermediate representations often reside in constrained spaces, such as the probability simplex for Softmax outputs or non-negative regions for ReLU activations. Standard KD typically focuses on final "soft labels," neglecting the rich structural information in hidden layers. Previous attempts to utilize these layers often use simple linear projections that ignore the specific transfer functions used. This paper introduces a geometrically principled approach using Bregman divergences to capture these non-Euclidean structures, facilitating more effective knowledge transfer.
Methods
The core of our approach is Extended Bregman PCA, which generalizes classical PCA in three ways:
- Bregman Divergences: We replace squared Euclidean distance with divergences tailored to the network's convex transfer functions (e.g., KL divergence for Softmax).
- Generalized Mean: We incorporate a mean vector ($m$) to handle non-centered activation data effectively.
- Riemannian Geometry: We redefine orthonormality using the Riemannian metric (the Hessian of the convex function), ensuring that compression directions respect the local curvature of the activation space.
Bregman Representation Learning (BRL)
To distill a model, we first perform Bregman PCA on a teacher's layer to identify principal directions ($V$) and a mean ($m$). These are fixed as a static layer in the student. The student network is then trained to predict the compression coefficients ($c_i$) for a given input. These coefficients are passed through the fixed teacher-derived layer to reconstruct the high-dimensional representation, as illustrated in Figure 1.
Results
We evaluated BRL on CIFAR-10, CIFAR-100, and ImageNet-1k. Our method achieves significantly lower reconstruction error compared to vanilla PCA. For instance, on CIFAR-100 output probabilities, Bregman PCA achieved a KL divergence loss several orders of magnitude lower than Euclidean baselines.
Representation Visualization
Visualizations using TriMap (Figure 2) demonstrate that representations in the penultimate layer form clusters similar to the final output. Bregman PCA captures this clustering structure with high fidelity, even at low dimensions ($k$).
Knowledge Distillation Performance
BRL proves exceptionally effective in low-data regimes. On CIFAR-10 with only 50% of training data, BRL achieved ~82% accuracy, a 9% improvement over standard soft-label distillation (~73%). On ImageNet, distilling a ResNet-50 into a ResNet-18 resulted in 72.6% Top-1 accuracy, outperforming standard KD (71.5%) and the baseline student (70.1%).
Discussion and Limitations
While BRL offers a powerful framework for representation learning, it currently requires strictly monotonic transfer functions (e.g., Leaky ReLU, Sigmoid). Standard ReLU layers, which have zero gradients in negative regions, present a challenge for the Bregman construction. Future work will explore multi-layer distillation and extending the framework to non-monotonic functions.
Summary updated