Reshaping Visual Datasets for Domain Adaptation

Boqing Gong, Kristen Grauman, Fei ShaNeurIPS 2013 · OpenReview

In visual recognition, domain adaptation is crucial for bridging the gap between training and testing data distributions. Standard approaches often assume that a single dataset represents a single, cohesive "domain." However, real-world data is rarely that neat; it is typically a messy mixture of varying lighting conditions, poses, and backgrounds.

This paper introduces a nonparametric approach to automatically discover "latent domains" within visual datasets. By reshaping datasets into these more coherent latent domains based on statistical distinctiveness and learnability, the authors achieve significantly better performance on object and action recognition tasks.

The Problem with Standard Dataset Boundaries

In machine learning, domain adaptation involves training a model on one distribution (the source) and applying it to another (the target). Traditionally, researchers treat an entire dataset—such as "images from Amazon"—as a single domain.

However, the authors argue this is a weak proxy for reality. A single dataset is often an agglomeration of many different conditions: some images have white backgrounds while others are cluttered, and some are high-resolution while others are blurry.

Visual variations within a single object category (e.g., keyboards on different backgrounds) illustrate why treating an entire dataset as one domain is problematic.

Treating this heterogeneous mix as one domain blurs the underlying statistics, making it harder for adaptation algorithms to find a clean mapping to a new target. The core challenge is automatically slicing these datasets into meaningful, discrete domains without manually labeling every attribute like "lighting" or "camera angle."

Methods: Discovering Latent Domains

The authors propose a framework to discover latent domains based on two axiomatic properties: Maximum Distinctiveness and Maximum Learnability.

1. Maximum Distinctiveness: The discovered domains must be as statistically different from each other as possible. To measure this without assuming a specific distribution (like a Gaussian), the authors use a kernel-based method in a Reproducing Kernel Hilbert Space (RKHS). Specifically, they maximize the distance between the means of the empirical distributions of the domains.

2. Maximum Learnability: Distinctiveness alone is insufficient, as one could easily create "distinct" domains composed of random noise. To ensure the domains are practically useful, the authors employ Domain-Wise Cross-Validation (DWCV). They choose the number of domains ($K$) that yields the highest cross-validation accuracy, ensuring each domain can actually support a strong discriminative model.

Domain-Wise Cross-Validation (DWCV) is used to determine the optimal number of latent domains by maximizing cross-validation accuracy.

The Optimization Process: The problem is initially formulated as an integer program (assigning each image to a domain), which is NP-hard. The authors relax this into a continuous quadratic programming problem. They also introduce a Label Prior Constraint (LPC), which ensures that the relative proportions of object categories (e.g., the ratio of cars to planes) remain consistent across the newly formed domains. This prevents the algorithm from simply clustering images by their object category.

Key Insights and Experimental Results

The approach was validated on two primary benchmarks: the "Office-Caltech" collection for object recognition (featuring datasets like Amazon, DSLR, Webcam, and Caltech-256) and the IXMAS multi-view dataset for human activity recognition.

  • Dataset Boundaries are Arbitrary: The labels given by dataset collectors are often less relevant to a model than latent factors like "side-view" or "dark lighting." A nonparametric approach, unlike previous attempts using Gaussian Mixture Models, doesn't force the data into a specific shape, making it much more robust for complex visual data.
  • Reshaping the Test Set: Most domain adaptation focuses solely on the training data. This paper shows that if the test set is also heterogeneous, performing "conditional reshaping"—splitting the test data into subsets that match the discovered training domains—can further boost accuracy. On cross-view action recognition, this conditional reshaping provided a consistent lift, improving accuracy from 52.3% to 54.9% in one multi-view configuration.
  • Oracle and Practical Accuracy: In "best-case" scenarios where the target domain is known, the discovered latent domains consistently outperformed the original dataset boundaries. On the IXMAS dataset, using discovered domains pushed accuracy from 47.1% to 50.3%. Furthermore, by using a "Matching" strategy (automatically picking the best source domain for a given target), the authors achieved results nearly identical to the Oracle performance, even without knowing the target labels.

Limitations and Future Work

While highly effective, the approach has limitations. The relaxation to quadratic programming makes the problem solvable, but kernel matrices scale quadratically with the number of instances, which could become a computational bottleneck for massive datasets. Additionally, while the algorithm finds "better" domains, it doesn't explicitly interpret why they are better (e.g., it won't explicitly label a domain as "low-light images"). Future work could involve examining the discovered domains to better understand these dominant factors of variation, as well as tuning optimization parameters based on specific visual features.

Summary updated

Abstract

In visual recognition problems, the common data distribution mismatches between training and testing make domain adaptation essential. However, image data is difficult to manually divide into the discrete domains required by adaptation algorithms, and the standard practice of equating datasets with domains is a weak proxy for all the real conditions that alter the statistics in complex ways (lighting, pose, background, resolution, etc.) We propose an approach to automatically discover latent domains in image or video datasets. Our formulation imposes two key properties on domains: maximum distinctiveness and maximum learnability. By maximum distinctiveness, we require the underlying distributions of the identified domains to be different from each other; by maximum learnability, we ensure that a strong discriminative model can be learned from the domain. We devise a nonparametric representation and efficient optimization procedure for distinctiveness, which, when coupled with our learnability constraint, can successfully discover domains among both training and test data. We extensively evaluate our approach on object recognition and human activity recognition tasks.
BibTeX
@inproceedings{gong2013reshaping,
  author = {Boqing Gong and Kristen Grauman and Fei Sha},
  title = {{Reshaping Visual Datasets for Domain Adaptation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2013},
  url = {https://openreview.net/forum?id=SJEqjP-_-r},
}
Figure