Multi-class Heterogeneous Domain Adaptation

Joey Tianyi Zhou, Ivor W. Tsang, Sinno Jialin Pan, Mingkui TanJMLR 2019 · OpenReview

TL;DR

This paper introduces Sparse Heterogeneous Feature Representation (SHFR), a novel method for multi-class Heterogeneous Domain Adaptation (HDA). SHFR learns a sparse feature mapping between domains with different feature types by framing the problem as a compressed sensing task, leveraging error-correcting codes for robustness and efficiency. It achieves state-of-the-art performance and significantly improves training speed compared to existing HDA methods.

What Problem Does This Paper Address?

Machine learning models often struggle when applied to data from a different domain than they were trained on, especially when the data's feature representations are fundamentally different (e.g., text vs. images, or different languages). This is known as domain adaptation. A more challenging scenario, Heterogeneous Domain Adaptation (HDA), arises when the feature spaces themselves are different (e.g., different dimensionality, different feature types). This paper tackles HDA in the context of multi-class classification, where existing methods often struggle with:

  • Dense Feature Mappings: Recovering dense mappings can lead to loss of interpretability and introduce noise, impacting classification accuracy.
  • Multi-class Handling: Many methods adapt binary classifiers independently, failing to leverage the structure of multi-class problems.
  • High Computational Cost: Estimating feature mappings can be computationally prohibitive, especially for high-dimensional data.

The authors aim to develop an efficient and effective HDA method that can learn a sparse, interpretable, and robust feature transformation for multi-class problems.

Methods

The core of SHFR lies in its novel approach to learning a feature mapping $G$ between a source domain (with features $X_s$) and a target domain (with features $X_t$).

  • Sparse Feature Representation: The key assumption is that the feature mapping $G$ is row-sparse. This means each feature in the target domain can be represented by a small subset of features from the source domain, inspired by how words translate between languages. This sparsity is enforced using an $l_1$-norm regularization.
  • Class-Invariance: The method assumes the feature mapping $G$ is class-invariant, meaning it's the same for all classes. This is achieved by decomposing the multi-class problem into multiple binary classification tasks and jointly optimizing $G$ across all these tasks.
  • Compressed Sensing Formulation: The problem of learning $G$ is framed as a compressed sensing (CS) problem. The weight vectors of binary classifiers learned in the source and target domains act as "measurement sensors." The goal is to recover a sparse signal (the feature mapping $G$) from these measurements.
  • Error Correcting Output Codes (ECOC): To generate a sufficient number of "incoherent" binary classifiers (crucial for effective CS), the paper proposes using the ECOC scheme. This allows for more robust estimation of $G$, even if some individual binary classifiers are not perfectly accurate due to limited target domain data.
  • Batch-Mode Matching Pursuit LASSO (BMPL): To efficiently solve the resulting nonnegative sparse recovery problem, a batch-mode matching pursuit LASSO algorithm is developed. This significantly reduces computational cost, especially for high-dimensional source data.

Datasets

The paper evaluates SHFR on both synthetic and real-world datasets:

  • Synthetic Dataset: A 20-class toy dataset was generated to test the ability of SHFR to recover a ground-truth sparse feature mapping.
  • Real-world Datasets:
    • Multilingual Reuters Collection: A text dataset with over 11,000 articles in six categories and five languages. Spanish was used as the target domain, with other languages as source domains.
  • BBC Collection: A multi-view text dataset where different "views" of articles were treated as different domains.
  • Cross-lingual Sentiment Dataset: A dataset of product reviews in four languages, used for cross-lingual sentiment classification.

Key Insights and Ideas

  • Sparsity is Key for HDA: The paper strongly advocates for sparsity in the feature mapping $G$ as a crucial constraint for effective HDA. It not only reduces computational cost but also improves the accuracy of the learned mapping.
  • ECOC Enhances Robustness: Using ECOC to generate binary classifiers is vital. It provides enough "measurements" for the compressed sensing formulation and makes the overall HDA system more robust to inaccuracies in individual target domain classifiers, which is common with limited labeled data.
  • Class-Invariance and Sparsity Complement Each Other: By jointly learning a class-invariant and sparse mapping $G$, SHFR effectively leverages common structures across classes and domains, leading to better generalization.
  • Compressed Sensing as a Powerful Framework for HDA: The paper demonstrates that viewing HDA as a compressed sensing problem, where classifier weights are measurements, is a highly effective approach.

Key Results

  • Superior Accuracy: SHFR consistently outperformed state-of-the-art HDA methods (DAMA, ARC-t, HFA) across all tested real-world datasets, achieving significant improvements in classification accuracy. For example, on the Reuters dataset, SHFR achieved accuracies in the low 70s, while baselines were in the high 60s.
  • Efficient Training: SHFR demonstrated significantly faster training times compared to baselines, especially for high-dimensional data. Its computational complexity is largely independent of source domain dimensionality and scales linearly with target domain dimensionality.
  • Accurate Feature Mapping Recovery: On synthetic data, SHFR was able to perfectly recover the sparse ground-truth feature mapping $G$ with little noise, unlike other methods that produced dense, noisy mappings.
  • Robustness to Limited Data: SHFR showed strong performance even with a small number of labeled target domain instances, outperforming baselines when target data was scarce.
  • Impact of ECOC: Using ECOC for binary classifiers led to more stable and superior performance compared to one-vs-all or one-vs-one strategies, especially when labeled data was limited.

Limitations and Future Work

  • Nonlinear Feature Transformation: The current SHFR method primarily focuses on linear feature mappings. The authors suggest exploring nonlinear feature transformations for image and text representations as a future direction.
  • Negative Alignment: While SHFR generally performs well, the paper notes that "negative transfer" (where alignment between source and target classifiers is detrimental) can occur, particularly when initial target classifiers are very weak. While ECOC helps mitigate this, further investigation into removing or handling such negative alignments is suggested.
  • Dichotomizer Size: The paper observes that while more dichotomizers generally improve performance, there's a point of diminishing returns where redundancy can hinder progress. Further research into optimal ECOC coding matrix design could be beneficial.

Summary updated

Abstract

A crucial issue in heterogeneous domain adaptation (HDA) is the ability to learn a feature mapping between different types of features across domains. Inspired by language translation, a word translated from one language corresponds to only a few words in another language, we present an efficient method named Sparse Heterogeneous Feature Representation (SHFR) in this paper for multi-class HDA to learn a sparse feature transformation between domains with multiple classes. Specifically, we formulate the problem of learning the feature transformation as a compressed sensing problem by building multiple binary classifiers in the target domain as various measurement sensors, which are decomposed from the target multi-class classification problem. We show that the estimation error of the learned transformation decreases with the increasing number of binary classifiers. In other words, for adaptation across heterogeneous domains to be successful, it is necessary to construct a sufficient number of incoherent binary classifiers from the original multi-class classification problem. To achieve this, we propose to apply the error correcting output correcting (ECOC) scheme to generate incoherent classifiers. To speed up the learning of the feature transformation across domains, we apply an efficient batch-mode algorithm to solve the resultant nonnegative sparse recovery problem. Theoretically, we present a generalization error bound of our proposed HDA method under a multi-class setting. Lastly, we conduct extensive experiments on both synthetic and real-world datasets to demonstrate the superiority of our proposed method over existing state-of-the-art HDA methods in terms of prediction accuracy and training efficiency.
BibTeX
@article{zhou2019multi,
  author = {Joey Tianyi Zhou and Ivor W. Tsang and Sinno Jialin Pan and Mingkui Tan},
  title = {{Multi-class Heterogeneous Domain Adaptation}},
  journal = {J. Mach. Learn. Res.},
  year = {2019},
  url = {https://openreview.net/forum?id=fL54TunbJMC},
}
Figure