Layerwise Bregman Representation Learning with Applications to Knowledge Distillation

Ehsan Amid, Rohan Anil, Christopher Fifty, Manfred K. Warmuth

Abstract

Knowledge distillation (KD) aims to transfer information from a complex teacher model to a simpler student model. Traditional methods often rely on Euclidean metrics for representation matching, which fail to account for the non-flat geometry of neural network activations. We propose Extended Bregman PCA, a dimensionality reduction technique that utilizes Bregman divergences to respect the intrinsic geometry of activation functions. By integrating this into a distillation framework called Bregman Representation Learning (BRL), we enable students to learn teacher-derived coordinate systems. Our results show that BRL significantly improves performance and sample efficiency across CIFAR and ImageNet benchmarks, particularly when training data is limited.

Introduction

In deep learning, intermediate representations often reside in constrained spaces, such as the probability simplex for Softmax outputs or non-negative regions for ReLU activations. Standard KD typically focuses on final "soft labels," neglecting the rich structural information in hidden layers. Previous attempts to utilize these layers often use simple linear projections that ignore the specific transfer functions used. This paper introduces a geometrically principled approach using Bregman divergences to capture these non-Euclidean structures, facilitating more effective knowledge transfer.

Methods

The core of our approach is Extended Bregman PCA, which generalizes classical PCA in three ways:

  1. Bregman Divergences: We replace squared Euclidean distance with divergences tailored to the network's convex transfer functions (e.g., KL divergence for Softmax).
  2. Generalized Mean: We incorporate a mean vector ($m$) to handle non-centered activation data effectively.
  3. Riemannian Geometry: We redefine orthonormality using the Riemannian metric (the Hessian of the convex function), ensuring that compression directions respect the local curvature of the activation space.

Bregman Representation Learning (BRL)

To distill a model, we first perform Bregman PCA on a teacher's layer to identify principal directions ($V$) and a mean ($m$). These are fixed as a static layer in the student. The student network is then trained to predict the compression coefficients ($c_i$) for a given input. These coefficients are passed through the fixed teacher-derived layer to reconstruct the high-dimensional representation, as illustrated in Figure 1.

Bregman divergence geometry and the BRL distillation framework.

Results

We evaluated BRL on CIFAR-10, CIFAR-100, and ImageNet-1k. Our method achieves significantly lower reconstruction error compared to vanilla PCA. For instance, on CIFAR-100 output probabilities, Bregman PCA achieved a KL divergence loss several orders of magnitude lower than Euclidean baselines.

Representation Visualization

Visualizations using TriMap (Figure 2) demonstrate that representations in the penultimate layer form clusters similar to the final output. Bregman PCA captures this clustering structure with high fidelity, even at low dimensions ($k$).

Visualization of CIFAR-10 representations and their reconstructions using Bregman PCA.

Knowledge Distillation Performance

BRL proves exceptionally effective in low-data regimes. On CIFAR-10 with only 50% of training data, BRL achieved ~82% accuracy, a 9% improvement over standard soft-label distillation (~73%). On ImageNet, distilling a ResNet-50 into a ResNet-18 resulted in 72.6% Top-1 accuracy, outperforming standard KD (71.5%) and the baseline student (70.1%).

Discussion and Limitations

While BRL offers a powerful framework for representation learning, it currently requires strictly monotonic transfer functions (e.g., Leaky ReLU, Sigmoid). Standard ReLU layers, which have zero gradients in negative regions, present a challenge for the Bregman construction. Future work will explore multi-layer distillation and extending the framework to non-monotonic functions.

Summary updated

Abstract

In this work, we propose a novel approach for layerwise representation learning of a trained neural network. In particular, we form a Bregman divergence based on the layer's transfer function and construct an extension of the original Bregman PCA formulation by incorporating a mean vector and normalizing the principal directions with respect to the geometry of the local convex function around the mean. This generalization allows exporting the learned representation as a fixed layer with a non-linearity. As an application to knowledge distillation, we cast the learning problem for the student network as predicting the compression coefficients of the teacher's representations, which are passed as the input to the imported layer. Our empirical findings indicate that our approach is substantially more effective for transferring information between networks than typical teacher-student training using the teacher's penultimate layer representations and soft labels.
BibTeX
@misc{amid2022layerwise,
  author = {Ehsan Amid and Rohan Anil and Christopher Fifty and Manfred K. Warmuth},
  title = {{Layerwise Bregman Representation Learning with Applications to Knowledge Distillation}},
  year = {2022},
  eprint = {2209.07080},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2209.07080},
}
Figure