Introduction
Domain Generalization (DG) is the ability of a machine learning model to maintain performance on unseen, out-of-distribution (OOD) data. While high-capacity models have seen significant progress in DG, "tiny" neural networks—those constrained to less than 1MB of memory for IoT and mobile deployment—face a unique "capacity mismatch." These models lack the parameters necessary to learn the complex, invariant features that larger models utilize for robustness.
Research indicates that existing state-of-the-art DG techniques provide negligible benefits to tiny networks. Furthermore, the performance gap between large "teacher" models and tiny "student" models widens significantly on OOD data compared to standard in-distribution (ID) data. This paper introduces Out-of-distribution Knowledge Distillation (OKD), a training framework designed to bridge this gap without increasing the final model's parameter count.
Methodology: Out-of-distribution Knowledge Distillation (OKD)
To address the capacity limitations of tiny models, the authors propose shifting the focus from complex regularization to guided learning. OKD builds upon standard Knowledge Distillation (KD) by introducing a training objective that forces the student to mimic the teacher's behavior specifically on synthetic OOD data.
Disruptive Data Augmentation
To generate synthetic OOD data without external datasets, the framework employs "disruptive" augmentations. For image tasks, a combination of CutMix (patch-based pasting) and Mixup (linear blending) is used. For speech, Masking (zeroing audio segments) is applied. These methods are chosen because they preserve global structures while altering local statistics, providing the teacher model with enough context to offer meaningful guidance to the student.
Experimental Setup
The researchers evaluated OKD across several benchmarks:
- DOSCO (DOmain Shifts in COntext): A new suite of seven datasets where domain labels are generated by clustering images based on visual context (background, style, viewpoint) using a Vision Transformer.
- Standard DG Benchmarks: PACS and OfficeHome.
- Speech Recognition: A DG variant of the Google Speech Commands dataset, partitioned by non-overlapping speakers to simulate domain shift.
Key Results
Experimental results demonstrate that OKD consistently outperforms both standard KD and traditional DG algorithms like MixStyle and RSC.
On the DOSCO-2k benchmark, OKD improved average OOD accuracy by 3% over standard KD. In speech recognition tasks, the improvement was even more pronounced, with OKD outperforming the Empirical Risk Minimization (ERM) baseline by 11% and standard KD by 3%. Crucially, because OKD only modifies the training loss, the final deployed model retains its original size and inference speed, providing "free" robustness for on-device applications.
Conclusion and Future Work
OKD represents a significant step toward making tiny models robust enough for real-world deployment. By leveraging the superior OOD handling of large models during training, tiny models can capture more generalizable features. However, a substantial performance gap remains—approximately 10% on DOSCO and 24% in speech tasks. Future research will explore on-device learning to update models post-deployment and learnable augmentation strategies to adaptively target a student model's specific weaknesses.
Summary updated