In traditional knowledge distillation, the relationship between models is strictly hierarchical: a high-capacity, pre-trained "teacher" provides static targets for a smaller "student" to mimic. While effective, this one-way flow assumes the teacher is infallible and that the student has nothing to offer in return.
Bidirectional knowledge distillation (often called mutual learning) reframes this process as a collaborative dialogue. Instead of a fixed teacher, two or more models—which may have different architectures, sizes, or data priorities—train simultaneously, using each other’s evolving predictions as soft labels. This peer-to-peer exchange allows models to transcend the performance caps of a static teacher and achieve better generalization.
The Mechanism of Peer Mimicry
The core of bidirectional distillation is the simultaneous optimization of multiple models. As Li et al. describe, this is typically achieved through a "cooperative mutual learning strategy" where models align their probability distributions using Kullback-Leibler (KL) divergence.
Unlike standard distillation, where the teacher's weights are frozen, both models in a mutual learning setup remain plastic. As one model discovers a more robust feature representation, it propagates that insight to its peer via the mimicry loss. This creates a dynamic regularization effect: the models are forced to reach a consensus, which smooths the feature space and prevents any single model from overfitting to idiosyncratic noise in the training data.
Why It Works: Complementary Perspectives
The primary intuition behind bidirectional distillation is that different models "see" the same data through different lenses. By exchanging knowledge, they can fill each other's blind spots.
- Structural Perspectives: In code generation, Xie et al. showed that a decoder using pre-order traversal and one using breadth-first traversal capture different structural dependencies. By teaching each other, they integrate both vertical and horizontal context. Similarly, Bian et al. used bidirectional learning between left-to-right and right-to-left decoders to improve mathematical expression recognition.
- Domain and Modality Perspectives: Chen et al. demonstrated that a model focused on linguistic patterns and one focused on Knowledge Graph structures can refine each other's labels. In speech translation, Zhao et al. found that a speech model and a text-based translation model acting as peers could improve each other, moving beyond the limits of a one-way cross-modal transfer.
- Data Priorities: In multilingual translation, Huang et al. proposed training two models with opposite sampling priorities—one favoring high-resource languages and the other low-resource. Through mutual distillation, the models exchange specialized knowledge, pushing the "Pareto frontier" of performance outward for all languages.
The Student as a Regularizer
One of the most surprising findings in this field is that a smaller student model can actually improve a larger teacher. Bhardwaj et al. argue that the limited capacity of a student model acts as a natural regularizer. Because the student lacks the parameters to memorize noise, its predictions are often "cleaner" than the teacher's. By forcing the teacher to match the student, the teacher is discouraged from memorizing corrupted labels or noise.
This effect is so potent that Jain and Gandhi found that even two pretrained models of similar size can improve each other by acting as mutual teachers, leveraging their unique architectural biases to fill predictive gaps.
Bridging the Cloud-Edge Gap
Bidirectional distillation has become a cornerstone of Federated Learning and Large-Small model collaboration. In these settings, the "teacher" (a large model on a server) and the "student" (a small model on a device) have a symbiotic relationship.
As Wang et al. and Fan et al. explain, the large model provides general reasoning capabilities to the edge devices, while the edge models distill specialized, local context back to the cloud. This feedback loop allows the global model to adapt to real-world user environments without requiring direct access to private raw data, ensuring that both the "brain" in the cloud and the "limbs" at the edge evolve together.
Key Foundational Work
KDGAN: Knowledge Distillation with Generative Adversarial Networks (2018) — contributed 2 concept elements to this direction.
Iterative Dual Domain Adaptation for Neural Machine Translation (2019) — contributed 1 concept elements to this direction.
Deeply-supervised Knowledge Synergy (2019) — contributed 1 concept elements to this direction.