TL;DR
This paper introduces Sparse Heterogeneous Feature Representation (SHFR), a novel method for multi-class Heterogeneous Domain Adaptation (HDA). SHFR learns a sparse feature mapping between domains with different feature types by framing the problem as a compressed sensing task, leveraging error-correcting codes for robustness and efficiency. It achieves state-of-the-art performance and significantly improves training speed compared to existing HDA methods.
What Problem Does This Paper Address?
Machine learning models often struggle when applied to data from a different domain than they were trained on, especially when the data's feature representations are fundamentally different (e.g., text vs. images, or different languages). This is known as domain adaptation. A more challenging scenario, Heterogeneous Domain Adaptation (HDA), arises when the feature spaces themselves are different (e.g., different dimensionality, different feature types). This paper tackles HDA in the context of multi-class classification, where existing methods often struggle with:
- Dense Feature Mappings: Recovering dense mappings can lead to loss of interpretability and introduce noise, impacting classification accuracy.
- Multi-class Handling: Many methods adapt binary classifiers independently, failing to leverage the structure of multi-class problems.
- High Computational Cost: Estimating feature mappings can be computationally prohibitive, especially for high-dimensional data.
The authors aim to develop an efficient and effective HDA method that can learn a sparse, interpretable, and robust feature transformation for multi-class problems.
Methods
The core of SHFR lies in its novel approach to learning a feature mapping $G$ between a source domain (with features $X_s$) and a target domain (with features $X_t$).
- Sparse Feature Representation: The key assumption is that the feature mapping $G$ is row-sparse. This means each feature in the target domain can be represented by a small subset of features from the source domain, inspired by how words translate between languages. This sparsity is enforced using an $l_1$-norm regularization.
- Class-Invariance: The method assumes the feature mapping $G$ is class-invariant, meaning it's the same for all classes. This is achieved by decomposing the multi-class problem into multiple binary classification tasks and jointly optimizing $G$ across all these tasks.
- Compressed Sensing Formulation: The problem of learning $G$ is framed as a compressed sensing (CS) problem. The weight vectors of binary classifiers learned in the source and target domains act as "measurement sensors." The goal is to recover a sparse signal (the feature mapping $G$) from these measurements.
- Error Correcting Output Codes (ECOC): To generate a sufficient number of "incoherent" binary classifiers (crucial for effective CS), the paper proposes using the ECOC scheme. This allows for more robust estimation of $G$, even if some individual binary classifiers are not perfectly accurate due to limited target domain data.
- Batch-Mode Matching Pursuit LASSO (BMPL): To efficiently solve the resulting nonnegative sparse recovery problem, a batch-mode matching pursuit LASSO algorithm is developed. This significantly reduces computational cost, especially for high-dimensional source data.
Datasets
The paper evaluates SHFR on both synthetic and real-world datasets:
- Synthetic Dataset: A 20-class toy dataset was generated to test the ability of SHFR to recover a ground-truth sparse feature mapping.
- Real-world Datasets:
- Multilingual Reuters Collection: A text dataset with over 11,000 articles in six categories and five languages. Spanish was used as the target domain, with other languages as source domains.
- BBC Collection: A multi-view text dataset where different "views" of articles were treated as different domains.
- Cross-lingual Sentiment Dataset: A dataset of product reviews in four languages, used for cross-lingual sentiment classification.
Key Insights and Ideas
- Sparsity is Key for HDA: The paper strongly advocates for sparsity in the feature mapping $G$ as a crucial constraint for effective HDA. It not only reduces computational cost but also improves the accuracy of the learned mapping.
- ECOC Enhances Robustness: Using ECOC to generate binary classifiers is vital. It provides enough "measurements" for the compressed sensing formulation and makes the overall HDA system more robust to inaccuracies in individual target domain classifiers, which is common with limited labeled data.
- Class-Invariance and Sparsity Complement Each Other: By jointly learning a class-invariant and sparse mapping $G$, SHFR effectively leverages common structures across classes and domains, leading to better generalization.
- Compressed Sensing as a Powerful Framework for HDA: The paper demonstrates that viewing HDA as a compressed sensing problem, where classifier weights are measurements, is a highly effective approach.
Key Results
- Superior Accuracy: SHFR consistently outperformed state-of-the-art HDA methods (DAMA, ARC-t, HFA) across all tested real-world datasets, achieving significant improvements in classification accuracy. For example, on the Reuters dataset, SHFR achieved accuracies in the low 70s, while baselines were in the high 60s.
- Efficient Training: SHFR demonstrated significantly faster training times compared to baselines, especially for high-dimensional data. Its computational complexity is largely independent of source domain dimensionality and scales linearly with target domain dimensionality.
- Accurate Feature Mapping Recovery: On synthetic data, SHFR was able to perfectly recover the sparse ground-truth feature mapping $G$ with little noise, unlike other methods that produced dense, noisy mappings.
- Robustness to Limited Data: SHFR showed strong performance even with a small number of labeled target domain instances, outperforming baselines when target data was scarce.
- Impact of ECOC: Using ECOC for binary classifiers led to more stable and superior performance compared to one-vs-all or one-vs-one strategies, especially when labeled data was limited.
Limitations and Future Work
- Nonlinear Feature Transformation: The current SHFR method primarily focuses on linear feature mappings. The authors suggest exploring nonlinear feature transformations for image and text representations as a future direction.
- Negative Alignment: While SHFR generally performs well, the paper notes that "negative transfer" (where alignment between source and target classifiers is detrimental) can occur, particularly when initial target classifiers are very weak. While ECOC helps mitigate this, further investigation into removing or handling such negative alignments is suggested.
- Dichotomizer Size: The paper observes that while more dichotomizers generally improve performance, there's a point of diminishing returns where redundancy can hinder progress. Further research into optimal ECOC coding matrix design could be beneficial.
Summary updated