Domain Adaptation with Ensemble of Feature Groups

Rajhans Samdani, Wen-tau YihIJCAI 2011 · OpenReview

Abstract

Domain adaptation is a critical challenge in machine learning where the distribution of training (source) data differs from testing (target) data. We present FEAD (Feature Ensembles for Domain Adaptation), a novel approach that leverages feature grouping and re-weighting. By training an ensemble of classifiers on distinct feature groups and tuning their weights using a small amount of target data, FEAD effectively prioritizes stable features over those prone to domain shift. Theoretical analysis and experimental results on sentiment classification and spam detection demonstrate that FEAD significantly outperforms state-of-the-art baselines.

Introduction

Standard machine learning assumes that training and testing data are independent and identically distributed (i.i.d.). However, in real-world scenarios like sentiment analysis or spam detection, this assumption often fails. For instance, the "language of praise" used in book reviews may differ from that in electronics reviews, or spammers may adapt their email content while their metadata remains relatively stable. Existing domain adaptation methods typically focus on instance re-weighting or finding shared feature spaces. FEAD addresses this by recognizing that different feature groups exhibit varying levels of stability across domains.

The FEAD Algorithm

The FEAD algorithm operates through a three-step process designed to provide a more "surgical" adaptation than single-parameter models:

  1. Feature Grouping: Features are partitioned into groups based on domain knowledge (e.g., content vs. metadata) or statistical stability measures.
  2. Individual Training: A separate classifier, such as Logistic Regression, is trained for each feature group using source domain data.
  3. Ensemble Weighting: The final model is a weighted ensemble of these classifiers. The weights ($\alpha$) are optimized using a small set of labeled target domain data.

Theoretically, this approach utilizes the $d_H$-distance framework to prove that ensemble models offer additional degrees of freedom. This allows the model to selectively down-weight or ignore feature groups with high divergence between source and target distributions.

Experimental Evaluation

The researchers evaluated FEAD on two primary tasks: sentiment classification using the Amazon benchmark dataset and large-scale email spam detection.

Sentiment Classification

FEAD was tested across 20 source-target pairs involving Books (B), DVDs (D), Electronics (E), Kitchen (K), and Movies (M). As shown in the results table, FEAD consistently outperformed the baseline Logistic Regression (LR) and other state-of-the-art methods like EasyAdapt (EA) and Multi-view Transfer Learning (Multi-T).

Table 1: Accuracy results for sentiment classification across 20 source-target domain pairs. FEAD achieves the highest average accuracy of 79.55%.

Spam Detection

In spam detection, "content" features are often unstable as spammers actively attempt to bypass filters, whereas "sender" features remain more consistent. FEAD's ability to re-weight these groups is particularly effective in the low false-positive region, which is critical for avoiding the misclassification of legitimate emails.

Figure 1: ROC curves in the low false-positive region for the time-shifted spam detection task. FEAD demonstrates superior performance compared to EA and LR.

In the "time-shifted" scenario, FEAD achieved a normalized Area Under the Curve (AUC) of 0.677, significantly higher than the 0.592 achieved by standard Logistic Regression.

Conclusion and Future Work

FEAD provides a robust framework for domain adaptation by exploiting the varying stability of feature groups. While the current implementation relies on manual or simple heuristic grouping and requires a small amount of labeled target data, it offers a significant performance boost over traditional methods. Future research will focus on automating the discovery of optimal feature groupings and extending the method to unsupervised domain adaptation scenarios.

Summary updated

Abstract

We present a novel approach for domain adaptation based on feature grouping and re-weighting. Our algorithm operates by creating an ensemble of multiple classifiers, where each classifier is trained on one particular feature group. Faced with the distribution change involved in domain change, different feature groups exhibit different cross-domain prediction abilities. Herein, ensemble models provide us the flexibility of tuning the weights of corresponding classifiers in order to adapt to the new domain. Our approach is supported by a solid theoretical analysis based on the expressiveness of ensemble classifiers, which allows trading-off errors across source and target domains. Moreover, experimental results on sentiment classification and spam detection show that our approach not only outperforms the baseline method, but is also superior to other state-of-the-art methods.
BibTeX
@inproceedings{samdani2011domain,
  author = {Rajhans Samdani and Wen-tau Yih},
  title = {{Domain Adaptation with Ensemble of Feature Groups}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2011},
  url = {https://openreview.net/forum?id=ry-S-rGu-S},
}
Figure