Correlation Alignment for Unsupervised Domain Adaptation

Baochen Sun, Jiashi Feng, Kate Saenko

TL;DR

CORAL (CORrelation ALignment) is a "frustratingly easy" yet powerful unsupervised domain adaptation method that aligns the second-order statistics (covariances) of source and target distributions. By whitening source features and "re-coloring" them with the target's covariance, the model bridges the domain gap without requiring any labels in the target domain, achieving state-of-the-art results in both shallow and deep learning architectures.

What Problem Does This Paper Address?

In machine learning, we often assume that training data (source) and test data (target) come from the same distribution. In reality, "domain shift" occurs—for example, a model trained on studio photos of products might fail when deployed on grainy, low-light smartphone images.

While supervised adaptation exists, labeling new data for every new environment is expensive and slow. Unsupervised Domain Adaptation (UDA) aims to solve this by leveraging unlabeled target data. Existing UDA methods often rely on complex subspace manifolds or projection techniques that can be computationally expensive or lose vital information by reducing dimensionality. The authors sought a simpler, more robust way to align distributions directly in the original feature space.

Methods

The paper introduces three variations of the CORAL approach:

  1. Linear CORAL: This is the core algorithm. It applies a linear transformation to the source features. The process is intuitive:
  2. Whitening: It removes the correlations of the source domain (making the covariance matrix an identity matrix).
  3. Re-coloring: It injects the correlations of the target domain into the whitened source features.
  4. The result is a transformed source distribution that "looks" like the target distribution in terms of its feature correlations.
  5. CORAL-LDA: For scenarios where efficiency is critical (like object detection), the authors apply CORAL to Linear Discriminant Analysis. Instead of transforming every data point, they equivalently transform the classifier weights, making it much faster for high-dimensional data.
  6. Deep CORAL: This extends the idea to Deep Neural Networks (DNNs). The authors design a "CORAL loss" layer that calculates the Frobenius norm of the difference between the source and target covariance matrices. By training the network with a joint loss (Classification Loss + CORAL Loss), the DNN learns features that are simultaneously discriminative for the task and invariant across domains.

Datasets

The authors evaluated their methods on several standard benchmarks:

  • Office-Caltech10: 10 categories across four domains (Amazon, DSLR, Webcam, and Caltech256) using SURF features.
  • Standard Office Dataset: 31 categories across three domains (Amazon, DSLR, Webcam) used for deep feature evaluation.
  • Virtual & Virtual-Gray: Synthetic datasets generated from 3D CAD models to test adaptation from "virtual" to "real" worlds.
  • PASCAL VOC 2010: Used to estimate baseline background statistics for object detection.

Key Insights and Ideas

  • "Frustratingly Easy" Effectiveness: The most surprising insight is that aligning just the second-order statistics (covariance) is often more effective than complex manifold alignment or higher-order moment matching.
  • Preserving Information: Unlike subspace methods that project data onto a lower-dimensional $k$-dimensional basis, CORAL works with the original features. This avoids the "hyperparameter headache" of choosing the right $k$ and prevents information loss.
  • Asymmetric Transformation is Key: The authors found that transforming the source to match the target is superior to whitening both domains. Whitening both (symmetric) ignores the unique structure of the target domain, whereas "bridging" the source to the target preserves the target's unlabelled structure.
  • Deep Equilibrium: In Deep CORAL, the classification loss and CORAL loss act as counterparts. The classification loss prevents the network from collapsing all data to a single point (which would technically result in zero CORAL loss), while the CORAL loss prevents the network from overfitting to source-specific features.

Key Results

  • Shallow Features: On the Office-Caltech10 benchmark, Linear CORAL achieved an average accuracy of 46.7%, outperforming established methods like TCA (45.7%) and Subspace Alignment (45.9%).
  • Deep Learning: Deep CORAL achieved an average accuracy of 72.1% on the standard Office dataset, surpassing other deep adaptation methods like DAN (71.3%) and DDC (70.6%).
  • Object Detection: CORAL-LDA showed massive gains in "virtual-to-real" adaptation. For example, when training on DSLR data to detect objects in Webcam images, Mean Average Precision (MAP) jumped from 37.7% (no adaptation) to 67.1% with CORAL.
  • Stability: The method proved highly stable across different regularization parameters ($\lambda$), making it easier to deploy in practice than methods that are sensitive to hyperparameter tuning.

Limitations and Future Work

  • Second-Order Only: CORAL only captures the mean and covariance. It may fail to align distributions that have identical second-order statistics but differ in higher-order moments (like skewness or kurtosis).
  • Assumption of Similar Means: The CORAL-LDA variant assumes the difference between positive and negative class means is similar across domains, which may not always hold.
  • Future Directions: The authors suggest that CORAL could be combined with other domain-alignment loss functions (like MMD or adversarial losses) to capture even more complex distribution structures.

Reference: Sun, B., Feng, J., & Saenko, K. (2016). Correlation Alignment for Unsupervised Domain Adaptation. arXiv:1612.01939v1 [cs.CV].

Summary updated

Abstract

In this chapter, we present CORrelation ALignment (CORAL), a simple yet effective method for unsupervised domainDomain Adaptation (DA) unsupervised domain adaptation, unsupervised DA adaptation. CORAL minimizes domain shiftDomain shift by aligning the second-order statistics of source and target distributions, without requiring any target labels. In contrast to subspace manifold methods, it aligns the original feature distributions of the source and target domains, rather than the bases of lower-dimensional subspaces. It is also much simpler than other distribution matching methods. CORAL performs remarkably well in extensive evaluations on standard benchmark datasets. We first describe a solution that applies a linear transformation to source features to align them with target features before classifier training. For linear classifiers, we propose to equivalently apply CORAL to the classifier weights, leading to added efficiency when the number of classifiers is small but the number and dimensionality of target examples are very high. The resulting CORAL Linear Discriminant Analysis (CORAL-LDA) Correlation Alignment (CORAL) CORAL linear discriminant analysis (CORAL-LDA) outperforms LDA by a large margin on standard domain adaptation benchmarks. Finally, we extend CORAL to learn a nonlinear transformation that aligns correlations of layer activations in deep neural networks (DNNs). The resulting Deep CORALCorrelation Alignment (CORAL) deep CORAL (D-CORAL) approach works seamlessly with DNNs and achieves state-of-the-art performance on standard benchmark datasets. Our code is available at: https://github.com/VisionLearningGroup/CORAL .
BibTeX
@inproceedings{sun2017correlation,
  author = {Baochen Sun and Jiashi Feng and Kate Saenko},
  title = {{Correlation Alignment for Unsupervised Domain Adaptation}},
  booktitle = {Domain Adaptation in Computer Vision Applications},
  year = {2017},
  url = {https://openreview.net/forum?id=se4_ECBCSAN},
}

Loading related content...

Figure