Interpretable Embedding Procedure Knowledge Transfer via Stacked Principal Component Analysis and Graph Neural Network

Seunghyun Lee, Byung Cheol SongAAAI 2021 · OpenReview

Abstract

Knowledge Distillation (KD) typically focuses on transferring static feature representations from a teacher to a student network. This paper introduces a novel framework, Interpretable Embedding Procedure Knowledge Transfer (IEPKT), which shifts the focus to the embedding procedure—the dynamic process of data transformation across layers. By leveraging Stacked Principal Component Analysis (SPCA) for interpretability and Message Passing Neural Networks (MPNN) for relational transfer, the proposed method achieves state-of-the-art performance, particularly in data-scarce scenarios.

Introduction

Deep learning models are increasingly deployed on resource-constrained devices, necessitating efficient compression techniques like Knowledge Distillation. Traditional KD methods often treat intermediate layers as static snapshots, failing to capture the underlying logic of how a teacher model structures data. IEPKT addresses this by distilling the "embedding procedure," representing the journey of high-dimensional inputs into low-dimensional semantic spaces as a series of evolving graphs.

Problem Statement

Most KD approaches focus on "what" the teacher sees (feature maps) rather than "how" it processes information. These static snapshots lack interpretability and provide limited guidance for the student model's learning trajectory. The challenge is to define a mathematically transferable and human-interpretable representation of the embedding process that captures inter-data relations.

Methodology

The IEPKT framework is built upon two technical pillars: Stacked PCA and Message Passing Neural Networks.

Stacked Principal Component Analysis (SPCA)

To analyze the embedding process, the authors employ a two-stage PCA approach. The first stage compresses high-dimensional feature maps, while the second stage further reduces dimensionality to extract inter-data relations, enabling 3D visualization of the embedding space. To maintain efficiency during mini-batch training, Incremental PCA (IPCA) is used to ensure principal components reflect the global dataset distribution.

Relational Transfer via MPNN

The embedding procedure is modeled as a sequence of graphs where nodes represent data points and edges represent their relationships. A Message Passing Neural Network (MPNN) is tasked with estimating the alteration of these relations between successive layers. This "relational knowledge" is then distilled into the student network.

The IEPKT framework uses Teacher and Student Distillation modules to transfer relational knowledge via SPCA and MPNN.

To balance the distillation objective with the primary task (e.g., classification), a gradient clipping technique is implemented, preventing the distillation loss from overwhelming the target task loss.

Key Insights and Visualization

A core contribution of this work is the interpretability provided by the 3D visualizations. By reducing the embedding space, the authors demonstrate how networks cluster data: early layers focus on low-level visual features like color and texture, while deeper layers transition toward semantic clustering (e.g., grouping all instances of a specific class).

The overall pipeline demonstrates how data relations evolve from low-level visual features to high-level semantic clusters across network layers.

The "aha moment" of this research is that the transition of data relations between layers is a powerful form of knowledge. Capturing how the teacher model reconciles visual differences into semantic similarities provides a superior learning signal for the student.

Experimental Results

The authors evaluated IEPKT across standard benchmarks including CIFAR10/100, TinyImageNet, CUB200-2011, and MIT-scene.

  • Performance Gains: On CIFAR100, the student network showed a 2.28% improvement over the baseline, outperforming established methods like Relational Knowledge Distillation (RKD).
  • Robustness to Data Scarcity: The method excels when training data is limited. At a 10% sampling rate on TinyImageNet, IEPKT achieved 45.01% accuracy, significantly higher than the next best competitor (42.18%).
  • Cross-Architecture Transfer: The approach remains effective even when teacher and student architectures differ significantly (e.g., WideResNet to MobileNet-V2), suggesting that embedding procedure knowledge is a universal property.

Conclusion

IEPKT represents a significant step toward more transparent and effective knowledge distillation. By focusing on the relational journey of data through a network, it provides a more robust transfer mechanism than static feature matching. Future work may explore the application of this framework to non-convolutional architectures, such as Transformers in natural language processing, and further optimize the computational overhead associated with graph-based relational modeling.

Summary updated

Abstract

Knowledge distillation (KD) is one of the most useful techniques for light-weight neural networks. Although neural networks have a clear purpose of embedding datasets into the low-dimensional space, the existing knowledge was quite far from this purpose and provided only limited information. We argue that good knowledge should be able to interpret the embedding procedure. This paper proposes a method of generating interpretable embedding procedure (IEP) knowledge based on principal component analysis, and distilling it based on a message passing neural network. Experimental results show that the student network trained by the proposed KD method improves 2.28% in the CIFAR100 dataset, which is a higher performance than the state-of-the-art (SOTA) method. We also demonstrate that the embedding procedure knowledge is interpretable via visualization of the proposed KD process. The implemented code is available at https://github.com/sseung0703/IEPKT.
BibTeX
@inproceedings{lee2021interpretable,
  author = {Seunghyun Lee and Byung Cheol Song},
  title = {{Interpretable Embedding Procedure Knowledge Transfer via Stacked Principal Component Analysis and Graph Neural Network}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=ewP3sFuNgE},
}
Figure