Abstract
Neural Architecture Search (NAS) has revolutionized the design of deep learning models, often employing Mutual Knowledge Distillation (MKD) to enhance search efficiency. However, standard MKD methods suffer from a "blind" teaching process where poorly performing models can propagate inaccurate knowledge, leading to collective failure. This paper introduces Performance-Aware Mutual Knowledge Distillation (PAMKD), a framework that incorporates a performance scrutiny mechanism to ensure only high-quality models act as teachers. By decoupling weights and utilizing Group-wise Relative Similarity (GRS), PAMKD significantly improves the quality and robustness of discovered architectures across CIFAR and ImageNet benchmarks.
Introduction
Neural Architecture Search (NAS) aims to automate the discovery of high-performance neural network structures. To accelerate this process, researchers frequently use Mutual Knowledge Distillation (MKD), where multiple models (learners) share knowledge—typically soft labels or feature representations—to improve one another.
Despite its benefits, the authors identify a critical flaw in existing MKD approaches: they lack a mechanism to evaluate the quality of the "teacher." This leads to a "vicious circle" or cascaded failure. If a model performs poorly, it generates low-quality knowledge. When other models are forced to learn from this data, their performance also degrades, making them even worse teachers. PAMKD addresses this by introducing a "scrutiny" filter, ensuring that knowledge transfer only occurs from superior models to inferior ones.
Methods
PAMKD introduces two primary innovations to the NAS pipeline: a tri-level optimization strategy for performance scrutiny and a relational distillation metric called Group-wise Relative Similarity (GRS).
1. Performance Scrutiny via Tri-level Optimization
To prevent models from converging to a trivial "lazy" solution where they simply mimic each other without learning the task, PAMKD utilizes a three-stage optimization process:
- Stage I (Independent Training): Each learner independently trains a set of evaluation weights ($V$) on the training data.
- Stage II (Scrutiny and Distillation): Models are evaluated on a validation set. A learner is permitted to teach another only if its validation loss is lower. The distilled knowledge is then used to update a separate set of training weights ($W$).
- Stage III (Architecture Update): The structural designs (architectures) are updated by minimizing validation loss based on the performance of the distilled $W$ weights.
2. Group-wise Relative Similarity (GRS)
Standard distillation often focuses on individual data points. PAMKD employs GRS, which requires students to mimic how a teacher perceives relationships between groups of images (at least four at a time). This captures the underlying "manifold structure" of the dataset, providing a more robust and transferable form of knowledge than simple class labels.
Key Insights: The "Bad Teacher" Problem
The most significant empirical finding is the proof of collective failure in standard MKD. Without scrutiny, the average performance of a group of learners can actually be worse than a single model training independently.
As shown in Figure 1, standard MKD (orange) often results in higher error rates for individual learners compared to independent training (dark blue). PAMKD (light blue) successfully breaks this cycle, consistently achieving the lowest error rates across all learners.
Key Results
PAMKD was integrated into several state-of-the-art NAS frameworks, including DARTS, P-DARTS, PC-DARTS, and PR-DARTS. The results demonstrate consistent improvements across multiple benchmarks:
- CIFAR-100: When applied to PR-DARTS, PAMKD reduced the test error from 16.48% to 16.05%.
- ImageNet: In PC-DARTS experiments, PAMKD achieved a top-1 error of 22.8%, significantly outperforming the baseline (24.2%) and other distillation methods like MKD (23.2%) and OKDDip (23.6%).
- Robustness: In search spaces specifically designed to cause performance collapse, PAMKD maintained significantly lower error rates than vanilla DARTS or standard MKD.
Conclusion
PAMKD provides a robust solution to the "blind teaching" problem in mutual knowledge distillation for NAS. By ensuring that only qualified models act as teachers and utilizing relational knowledge through GRS, the framework discovers higher-quality architectures. While the method introduces computational overhead by training multiple learners simultaneously, the significant gains in architecture quality and search robustness make it a powerful tool for automated model design. Future work will focus on reducing the memory and processing requirements of the multi-learner search phase.
Summary updated