On-Device Domain Generalization

Kaiyang Zhou, Yuanhan Zhang, Yuhang Zang, Jingkang Yang, Chen Change Loy, Ziwei Liu

Introduction

Domain Generalization (DG) is the ability of a machine learning model to maintain performance on unseen, out-of-distribution (OOD) data. While high-capacity models have seen significant progress in DG, "tiny" neural networks—those constrained to less than 1MB of memory for IoT and mobile deployment—face a unique "capacity mismatch." These models lack the parameters necessary to learn the complex, invariant features that larger models utilize for robustness.

Research indicates that existing state-of-the-art DG techniques provide negligible benefits to tiny networks. Furthermore, the performance gap between large "teacher" models and tiny "student" models widens significantly on OOD data compared to standard in-distribution (ID) data. This paper introduces Out-of-distribution Knowledge Distillation (OKD), a training framework designed to bridge this gap without increasing the final model's parameter count.

Methodology: Out-of-distribution Knowledge Distillation (OKD)

To address the capacity limitations of tiny models, the authors propose shifting the focus from complex regularization to guided learning. OKD builds upon standard Knowledge Distillation (KD) by introducing a training objective that forces the student to mimic the teacher's behavior specifically on synthetic OOD data.

The OKD framework: Synthetic OOD samples are generated from in-distribution data, and the student is trained to match the teacher's predictions via KL divergence.

Disruptive Data Augmentation

To generate synthetic OOD data without external datasets, the framework employs "disruptive" augmentations. For image tasks, a combination of CutMix (patch-based pasting) and Mixup (linear blending) is used. For speech, Masking (zeroing audio segments) is applied. These methods are chosen because they preserve global structures while altering local statistics, providing the teacher model with enough context to offer meaningful guidance to the student.

Experimental Setup

The researchers evaluated OKD across several benchmarks:

  • DOSCO (DOmain Shifts in COntext): A new suite of seven datasets where domain labels are generated by clustering images based on visual context (background, style, viewpoint) using a Vision Transformer.
  • Standard DG Benchmarks: PACS and OfficeHome.
  • Speech Recognition: A DG variant of the Google Speech Commands dataset, partitioned by non-overlapping speakers to simulate domain shift.

Key Results

Experimental results demonstrate that OKD consistently outperforms both standard KD and traditional DG algorithms like MixStyle and RSC.

Performance comparison: (a) OKD shows the highest improvement over the ERM baseline. (b) and (c) show that OKD significantly reduces the performance gap between the tiny student and the large teacher on both OOD and ID data.

On the DOSCO-2k benchmark, OKD improved average OOD accuracy by 3% over standard KD. In speech recognition tasks, the improvement was even more pronounced, with OKD outperforming the Empirical Risk Minimization (ERM) baseline by 11% and standard KD by 3%. Crucially, because OKD only modifies the training loss, the final deployed model retains its original size and inference speed, providing "free" robustness for on-device applications.

Conclusion and Future Work

OKD represents a significant step toward making tiny models robust enough for real-world deployment. By leveraging the superior OOD handling of large models during training, tiny models can capture more generalizable features. However, a substantial performance gap remains—approximately 10% on DOSCO and 24% in speech tasks. Future research will explore on-device learning to update models post-deployment and learnable augmentation strategies to adaptively target a student model's specific weaknesses.

Summary updated

Abstract

We present a systematic study of domain generalization (DG) for tiny neural networks, a problem that is critical to on-device machine learning applications but has been overlooked in the literature where research has been focused on large models only. Tiny neural networks have much fewer parameters and lower complexity, and thus should not be trained the same way as their large counterparts for DG applications. We find that knowledge distillation is a strong candidate for solving the problem: it outperforms state-of-the-art DG methods that were developed using large models with a large margin. Moreover, we observe that the teacher-student performance gap on test data with domain shift is bigger than that on in-distribution data. To improve DG for tiny neural networks without increasing the deployment cost, we propose a simple idea called out-of-distribution knowledge distillation (OKD), which aims to teach the student how the teacher handles (synthetic) out-of-distribution data and is proved to be a promising framework for tackling the problem. We also contribute a scalable method for creating DG datasets, called DOmain Shift in COntext (DOSCO), which can be applied to broad data at scale without much human effort. Code and models will be released.
BibTeX
@misc{zhou2022device,
  author = {Kaiyang Zhou and Yuanhan Zhang and Yuhang Zang and Jingkang Yang and Chen Change Loy and Ziwei Liu},
  title = {{On-Device Domain Generalization}},
  year = {2022},
  eprint = {2209.07521},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2209.07521},
}
Figure