Learning Factorized Representations for Open-Set Domain Adaptation

Mahsa Baktashmotlagh, Masoud Faraki, Tom Drummond, Mathieu Salzmann

The Challenge of Open-Set Environments

Traditional Domain Adaptation (DA) assumes a "closed-set" scenario where the source (training) and target (test) domains share the exact same set of categories. However, in real-world applications, this assumption is often brittle. For instance, a model trained to recognize office supplies and deployed in a warehouse will inevitably encounter objects it has never seen before.

Open-Set Domain Adaptation (OSDA) tackles this reality. The goal is to correctly classify samples from known classes while accurately labeling samples from unseen classes as "unknown." Previous attempts at OSDA were either computationally expensive—relying on iterative assignment—or utilized adversarial deep learning that struggled to clearly separate knowns from unknowns.

To address this, researchers introduce FRODA (Factorized Representations for Open-set Domain Adaptation). By factorizing data into a shared subspace for known classes and a private subspace for unknown ones, the model effectively identifies and discards outliers, outperforming previous methods in both accuracy and computational speed.

How FRODA Works: Subspace Factorization

The core idea behind FRODA is geometric factorization. The framework models the data using two low-dimensional subspaces:

  1. Shared Subspace ($V$): Captures features common to both source and target domains (representing the "known" classes).
  2. Private Subspace ($U$): Captures features specific to the target domain that don't align with the source (representing the "unknown" classes).

To prevent a target sample from being modeled as a confusing mixture of both subspaces, FRODA applies a group sparsity regularizer. This forces the optimization to make a strict choice: a sample should be reconstructed by either the shared subspace or the private one. This "either/or" intuition is crucial—an unknown object shouldn't have "a bit of known features"; it should be fundamentally different.

The authors also propose a discriminative extension, D-FRODA, which adds a linear classifier term using source labels during the representation learning phase. This ensures the shared subspace isn't just good at reconstruction, but also optimized for categorization.

At test time, inference is straightforward: a simple ratio of reconstruction coefficients determines if a sample is unknown. If it is better explained by the private subspace than the shared one, it is labeled "unknown."

Key Results and Performance

The authors evaluated FRODA on major benchmarks, including the dense cross-dataset BCIS (comprising images from Bing, Caltech256, ImageNet, and SUN) and the standard Office dataset, using pre-trained DeCAF7 features.

Accuracy comparison of FRODA against state-of-the-art methods on the BCIS dataset.
  • State-of-the-Art Accuracy: As detailed in the results above, D-FRODA achieved an average accuracy of 65.7% on the BCIS dataset. This significantly outperforms previous non-deep methods like ATI (50.2%) and edges out deep methods like AODA (65.4%).
  • Massive Speedup: By formulating the problem as a series of sub-problems solvable with standard techniques (like the Lagrange dual method or proximal gradient descent), FRODA achieved massive speedups. It is roughly an order of magnitude faster than the previous leading non-deep method (ATI), converging in ~2.5 seconds compared to ATI's 32 seconds on the same hardware.
  • Robustness: The model demonstrated high stability to its internal hyperparameters ($\alpha, \beta, \lambda$), reducing the need for exhaustive tuning on new datasets. It also showed resilience when the source data contained its own set of unknown classes (evaluated as D-FRODA-U).

Limitations and Future Directions

While highly effective, FRODA relies on pre-trained features (like DeCAF7), meaning its performance is ultimately capped by the quality of the initial embedding. Furthermore, the current framework uses linear subspaces and classifiers. Future work could explore non-linear manifolds or integrate this factorization directly into the end-to-end training of deep networks, rather than applying it as a post-processing step. Finally, finding ways to better exploit unknown source data could further improve the separation of target unknowns.

Summary updated

Abstract

Domain adaptation for visual recognition has undergone great progress in the past few years. Nevertheless, most existing methods work in the so-called closed-set scenario, assuming that the classes...
BibTeX
@inproceedings{baktashmotlagh2019learning,
  author = {Mahsa Baktashmotlagh and Masoud Faraki and Tom Drummond and Mathieu Salzmann},
  title = {{Learning Factorized Representations for Open-Set Domain Adaptation}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=SxbNbAoBse9},
}
Figure