Unsupervised Attribute Alignment for Characterizing Distribution Shift

Matthew Lyle Olson, Shusen Liu, Rushil Anirudh, Jayaraman J. Thiagarajan, Weng-Keen Wong, Peer-timo BremerDistShift @ NeurIPS 2021 · OpenReview

TL;DR

This paper introduces a novel framework for unsupervised attribute alignment to characterize distribution shifts between datasets. By jointly discovering and aligning "attributes" (meaningful semantic factors of variation), the method provides an interpretable way to understand how data distributions differ, which is crucial for applications like scientific modeling.

What Problem Does This Paper Address?

Machine learning models often assume data distributions remain constant over time. However, in real-world scenarios, this "distribution shift" can lead to performance degradation or outright failure. While many methods focus on making models robust to these shifts, they often treat the shift as noise. This paper addresses the critical need to understand and characterize these shifts in an interpretable way, especially in scientific applications where ML models are trained on simulated data and deployed on real-world data. Understanding the differences between simulated and real data can reveal gaps in scientific understanding. The core challenge is to identify the underlying "attributes" or generative factors that explain these differences and similarities between datasets, and to align these attributes across distributions.

Methods

The authors propose a joint framework that tackles both attribute discovery and attribute alignment simultaneously. They leverage a pre-trained generator network (specifically, a beta-VAE decoder) that maps a latent space to a data distribution. An "attribute" is defined as a direction in this latent space that corresponds to a meaningful semantic change in the generated data.

The core of their approach involves training a discriminator network. This discriminator takes pairs of generated data (one original, one perturbed along a learned attribute direction) and predicts:

  1. Which attribute direction was used for perturbation.
  2. The magnitude of the perturbation.

This training process jointly optimizes the attribute discovery (learning the directions $W$) and the alignment (ensuring these directions are comparable across different generators/distributions). To prevent trivial solutions and encourage disentanglement, they incorporate:

  • Sparsity regularization: Encourages attributes to be independent.
  • Spectral norm orthogonality loss: Ensures learned attribute directions are distinct.

A key modification for aligning distributions is sampling the perturbation magnitude $\epsilon$ from a unidirectional range ($U(0.1 \times \epsilon, \epsilon)$) instead of a bidirectional one, to avoid issues with attribute directionality flipping between datasets.

Datasets

The paper evaluates their framework on two main experimental setups:

  1. MNIST vs. Colored MNIST: MNIST is a standard dataset of handwritten digits. Colored MNIST is a variant where random colors are applied to the digits, creating a 3-channel image. This setup explores shifts related to color and potentially other semantic attributes.
  2. dSprites vs. Square dSprites: dSprites is a synthetic dataset with 5 independent attributes (location, scale, rotation, shape). The square dSprites variant is a subset that only contains square shapes. This setup focuses on shifts related to shape and other geometric attributes.

These datasets are chosen because their attributes are semantically identifiable, allowing for easier evaluation of the learned alignments.

Key Insights and Ideas

  • Joint Discovery and Alignment is Crucial: The paper argues that attribute discovery and alignment are not independent problems. Aligning attributes requires discovering them in a way that makes them comparable across distributions.
  • Interpretable Distribution Shift Characterization: The framework offers a more intuitive way to understand distribution shifts beyond just measuring performance drops. It allows users to see what has changed (e.g., color, orientation, shape) and how it has changed.
  • Unsupervised Discovery of Shared and Unique Attributes: The method can identify attributes that are common to both distributions (shared) and those that are specific to one distribution (unique), providing a richer characterization.
  • Importance of Regularization: The success of the method hinges on regularization techniques (sparsity, orthogonality) to prevent the learned attributes from collapsing into trivial or overly complex representations.

Key Results

The paper presents qualitative results demonstrating the effectiveness of their approach.

  • dSprites Experiment: The learned attributes successfully capture positional shifts (moving shapes between cardinal directions) and rotation.
  • MNIST vs. Colored MNIST Experiment: The learned attributes identify changes in digit class, digit orientation, and image intensity. Notably, one learned attribute for Colored MNIST, while appearing unusual, makes sense as an increase in overall RGB intensity.
  • The paper shows figures with examples of learned shared and unique attributes for both datasets, visually confirming the framework's ability to disentangle and align semantic factors.

Limitations and Future Work

  • Generator Quality Dependency: The quality of the learned attributes and alignment is dependent on the quality and properties of the pre-trained generative models used.
  • Assumption of Predictability: The current approach assumes that semantically meaningful attributes are more predictive for discovery. This assumption might not hold universally, suggesting a need for more general attribute search criteria.
  • Iterative Learning: The authors suggest exploring progressive learning where the number of attributes ($K$) is increased iteratively, potentially guiding human exploration of learned attributes.
  • Broader Applicability: While tested on MNIST and dSprites, the authors aim to apply this to more complex domains like faces vs. anime faces, or experimental scientific data vs. simulated scientific data.

Summary updated

Abstract

Detecting and addressing distribution shift is an important task in machine learning. However, most of the machine learning solutions to deal with distribution shift lack the capability to identify the key characteristics of such a shift and present it to humans in an interpretable way. In this work, we propose a novel framework to compare two datasets and identify distribution shifts between the datasets. The key challenge is to identify generative factors of variation, which we refer to as attributes, that characterize the similarities and differences between the datasets. Producing this characterization requires finding a set of attributes that can be aligned between the two datasets and sets that are unique. We address this challenge through a novel approach that performs both attribute discovery and attribute alignment across the two distributions. We evaluate our algorithm's effectiveness at accurately identifying these attributes in two separate experiments, one involving two variants of MNIST and a second experiment involving two versions of dSprites.
BibTeX
@inproceedings{olson2021unsupervised,
  author = {Matthew Lyle Olson and Shusen Liu and Rushil Anirudh and Jayaraman J. Thiagarajan and Weng-Keen Wong and Peer-timo Bremer},
  title = {{Unsupervised Attribute Alignment for Characterizing Distribution Shift}},
  booktitle = {DistShift @ NeurIPS},
  year = {2021},
  url = {https://openreview.net/forum?id=Bk1hklAuZyh},
}

Loading related content...

Figure