Distilling Knowledge Across Heterogeneous Model Architectures

208 concepts164 papers1 proposal

Knowledge distillation (KD) is traditionally framed as a structural alignment problem. To compress a massive "teacher" network into a nimble "student" model, conventional approaches often force the student's intermediate feature maps to mimic the teacher's internal representations. However, this introduces a fundamental constraint: the student and teacher must share compatible mathematical structures, layer depths, or hidden dimensions. When architectures are heterogeneous—such as transferring knowledge from a Transformer to an LSTM, or between models with entirely different tokenizers—this internal representation gap widens, making direct feature matching structurally impossible.

To bypass this limitation, a powerful paradigm shift has emerged: architecture-agnostic knowledge distillation. Instead of forcing a student to replicate how a teacher computes, these methods focus on transferring what the teacher has learned. By treating the teacher as a black box and distilling knowledge through universal mediums—such as output distributions, geometric relationships, or data-level interventions—this approach allows developers to decouple the choice of deployment hardware from the architecture of the original foundation model.

The Universality of Logit-Based Alignment

The simplest universal medium shared by any two classification or autoregressive models is the final output probability distribution, or logits. Logits represent a high-level semantic space that captures not just the correct answer, but the "dark knowledge"—the relative similarities and soft correlations between different classes or tokens.

Because logits exist at the very end of the computational pipeline, they are completely decoupled from the internal network topology. As shown by de Oliveira et al. in their work on Distilling HuBERT with LSTMs via Decoupled Knowledge Distillation, logit-based distillation allows a student model to optimize for its own architectural strengths (like an LSTM's sequential efficiency) while still absorbing a teacher's predictive patterns (like a Transformer's contextual nuances).

[Teacher: Transformer] ──> [Internal Features] ──> [Logits]
                                                      │  (Decoupled Alignment)
                                                      ▼
[Student: LSTM]        ──> [Internal Features] ──> [Logits]

Operating exclusively on final output scores eliminates the need for complex adapter layers or dimensionality-matching projections. In a single-author study, Bowen Zheng demonstrated in Rethinking Decoupled Knowledge Distillation: A Predictive Distribution Perspective that refining how these logit distributions are partitioned can unlock latent knowledge in the long tail of predictions without adding any structural parameters to the student. This allows a student to inherit complex decision-making logic even when its internal mathematical structure is fundamentally simpler, a phenomenon also leveraged by Ashutosh Adhikari et al. in Exploring the Limits of Simple Learners in Knowledge Distillation for Document Classification with DocBERT to compress massive language models into single-layer LSTMs.

Bridging Mismatches via Geometric Alignment

While logits provide an architecture-agnostic bridge, standard divergence metrics like Kullback-Leibler (KL) divergence assume that the teacher and student share identical output spaces. This assumption breaks down when models utilize different vocabularies or tokenization schemes, as the dimensions of their output vectors no longer align.

To solve this, researchers have turned to geometric distance metrics rooted in optimal transport theory. Instead of comparing outputs element-by-element, optimal transport views the student and teacher distributions as geometric shapes or probability masses, measuring the minimal "work" required to transform one distribution into the other.

Teacher Logits (Vocabulary A) ───┐
                                 ├──> [Optimal Transport / Sinkhorn Distance] ──> Loss
Student Logits (Vocabulary B) ───┘

For instance, Mo Zhu et al. introduced Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Models, demonstrating that optimal transport can act as a dimension-agnostic bridge. This allows effective knowledge transfer between language models with completely incompatible tokenizers. Similarly, Xiao Cui et al. utilized Sinkhorn distance minimization in SinKD: Sinkhorn Distance Minimization for Knowledge Distillation to capture global geometric relationships across diverse architectures, including encoder-only and decoder-only structures. By aligning the underlying geometry of the semantic space rather than individual coordinate dimensions, the distillation process remains entirely agnostic to both the internal layers and the specific tokenization formats of the models.

Distillation as a Data-Level Intervention

Another highly effective strategy for cross-architecture transfer is to shift the distillation mechanism entirely out of the model space and into the data space. In this framing, the teacher model is treated as an oracle or a high-quality data generator, and distillation is achieved by optimizing the training data itself to match the student's inductive biases.

This data-centric approach manifests in several ways:

By focusing on data-level interventions, the student model learns the underlying task logic and behavioral constraints directly from the curated data distribution, rendering the structural differences between the source and target models completely irrelevant.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 156 papers
  2. Version 1 · 2026-10-04 · 249 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_architecture_agnostic_distillation_5e353a1b,
  title = {architecture-agnostic distillation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/architecture-agnostic-distillation/rdr_5e353a1b6a1240949978d4106b6461f7},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_architecture_agnostic_distillation_5e353a1b_v2,
  title = {architecture-agnostic distillation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/architecture-agnostic-distillation/rdr_5e353a1b6a1240949978d4106b6461f7/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake