Distribution Shift Inversion for Out-of-Distribution Prediction

Runpeng Yu, Songhua Liu, Xingyi Yang, Xinchao Wang

Abstract

Machine learning society has witnessed the emergence of a myriad of Out-of-Distribution (OoD) algorithms, which address the distribution shift between the training and the testing distribution by searching for a unified predictor or invariant feature representation. However, the task of directly mitigating the distribution shift in the unseen testing set is rarely investigated, due to the unavailability of the testing distribution during the training phase and thus the impossibility of training a distribution translator mapping between the training and testing distribution. In this paper, we explore how to bypass the requirement of testing distribution for distribution translator training and make the distribution translation useful for OoD prediction. We propose a portable Distribution Shift Inversion (DSI) algorithm, in which, before being fed into the prediction model, the OoD testing samples are first linearly combined with additional Gaussian noise and then transferred back towards the training distribution using a diffusion model trained only on the source distribution. Experimental results show that our method provides a general performance gain when plugged into a wide range of commonly used OoD algorithms.

Introduction

Machine learning models are notoriously brittle when they encounter data that looks different from their training set—a problem known as Out-of-Distribution (OoD) generalization. For example, a model trained on photos of dogs might fail when asked to identify a stylized cartoon dog. Traditionally, researchers attempt to solve this by making the model itself more robust, usually by searching for "invariant features" (characteristics that stay the same across photos and cartoons). However, this is incredibly difficult because the "unseen" test data is unknown during training.

Instead of training a model to be robust to every possible scenario, Distribution Shift Inversion (DSI) transforms the testing data back toward the training distribution before the model ever sees it. By using diffusion models as "distribution magnets," DSI provides a plug-and-play performance boost for existing OoD algorithms without needing any prior knowledge of the testing domain.

Methodology

The core engine of DSI is a Diffusion Model trained exclusively on the source (training) distribution. Unlike standard domain adaptation, DSI requires zero target data during training. The process works in two main steps:

  1. Forward Transformation: An OoD test sample is mixed with a specific amount of Gaussian noise. This "blurs" the unique characteristics of the test domain, moving the sample into a "noise space" that the diffusion model understands.
  2. Backward Transformation: The diffusion model then "denoises" this mixture. Because the model was only trained on source data, it naturally pulls the sample toward the source distribution, effectively "inverting" the distribution shift while preserving semantic content.
The DSI pipeline: Out-of-Distribution (OoD) samples are transformed back toward the source distribution using a diffusion model and an iterative confidence-based schedule.

DSI also utilizes a Sample-Adaptive Schedule. Not all images need the same amount of transformation. The algorithm starts with a light transformation and checks the model's prediction confidence. If confidence is low, it applies a stronger transformation (more noise, more denoising) until a threshold is met or maximum steps are reached.

Experimental Results

The researchers evaluated DSI across several standard benchmarks, including PACS (Art, Cartoon, Photo, Sketch), OfficeHome, and ImageNet-R. DSI consistently improved performance across the board when integrated with existing algorithms like ERM, Mixup, and GroupDRO.

Experimental results on PACS and OfficeHome datasets showing consistent performance gains (Average Gain) when DSI is integrated with various baseline OoD algorithms.

Key results include:

  • General Gains: On average, adding DSI produced a 2.26% accuracy gain on multi-training domain datasets.
  • Robustness in Hard Domains: In the difficult "Sketch" domain of the PACS dataset, DSI improved average accuracy by 8.22%.
  • Spurious Correlations: On the CdSprites-5 dataset, DSI achieved a massive 33.95% average gain, effectively stripping away misleading environmental factors like background colors that confuse standard models.

Conclusion and Future Work

DSI represents a shift in perspective for OoD generalization: rather than building a more robust model, we can "fix" the data to match the model's expectations. The method requires zero knowledge of the target domain during training and is orthogonal to existing robust learning techniques.

However, limitations remain. Diffusion models are computationally expensive, which may limit real-time application. Additionally, aggressive transformations carry a risk of "hallucination," where the model might change the identity of the object entirely. Future work will focus on optimizing inference speed and refining confidence-based filtering to ensure label preservation.

Summary updated

Abstract

Machine learning society has witnessed the emergence of a myriad of Out-of-Distribution (OoD) algorithms, which address the distribution shift between the training and the testing distribution by searching for a unified predictor or invariant feature representation. However, the task of directly mitigating the distribution shift in the unseen testing set is rarely investigated, due to the unavailability of the testing distribution during the training phase and thus the impossibility of training a distribution translator mapping between the training and testing distribution. In this paper, we explore how to bypass the requirement of testing distribution for distribution translator training and make the distribution translation useful for OoD prediction. We propose a portable Distribution Shift Inversion (DSI) algorithm, in which, before being fed into the prediction model, the OoD testing samples are first linearly combined with additional Gaussian noise and then transferred back towards the training distribution using a diffusion model trained only on the source distribution. Theoretical analysis reveals the feasibility of our method. Experimental results, on both multiple-domain generalization datasets and single-domain generalization datasets, show that our method provides a general performance gain when plugged into a wide range of commonly used OoD algorithms. Our code is available at https://github.com/yu-rp/Distribution-Shift-Iverson.
BibTeX
@inproceedings{yu2023distribution,
  author = {Runpeng Yu and Songhua Liu and Xingyi Yang and Xinchao Wang},
  title = {{Distribution Shift Inversion for Out-of-Distribution Prediction}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  url = {https://openreview.net/forum?id=Yf5zUwpQcc},
}
Figure