Abstract
Domain adaptation is a critical challenge in machine learning where the distribution of training (source) data differs from testing (target) data. We present FEAD (Feature Ensembles for Domain Adaptation), a novel approach that leverages feature grouping and re-weighting. By training an ensemble of classifiers on distinct feature groups and tuning their weights using a small amount of target data, FEAD effectively prioritizes stable features over those prone to domain shift. Theoretical analysis and experimental results on sentiment classification and spam detection demonstrate that FEAD significantly outperforms state-of-the-art baselines.
Introduction
Standard machine learning assumes that training and testing data are independent and identically distributed (i.i.d.). However, in real-world scenarios like sentiment analysis or spam detection, this assumption often fails. For instance, the "language of praise" used in book reviews may differ from that in electronics reviews, or spammers may adapt their email content while their metadata remains relatively stable. Existing domain adaptation methods typically focus on instance re-weighting or finding shared feature spaces. FEAD addresses this by recognizing that different feature groups exhibit varying levels of stability across domains.
The FEAD Algorithm
The FEAD algorithm operates through a three-step process designed to provide a more "surgical" adaptation than single-parameter models:
- Feature Grouping: Features are partitioned into groups based on domain knowledge (e.g., content vs. metadata) or statistical stability measures.
- Individual Training: A separate classifier, such as Logistic Regression, is trained for each feature group using source domain data.
- Ensemble Weighting: The final model is a weighted ensemble of these classifiers. The weights ($\alpha$) are optimized using a small set of labeled target domain data.
Theoretically, this approach utilizes the $d_H$-distance framework to prove that ensemble models offer additional degrees of freedom. This allows the model to selectively down-weight or ignore feature groups with high divergence between source and target distributions.
Experimental Evaluation
The researchers evaluated FEAD on two primary tasks: sentiment classification using the Amazon benchmark dataset and large-scale email spam detection.
Sentiment Classification
FEAD was tested across 20 source-target pairs involving Books (B), DVDs (D), Electronics (E), Kitchen (K), and Movies (M). As shown in the results table, FEAD consistently outperformed the baseline Logistic Regression (LR) and other state-of-the-art methods like EasyAdapt (EA) and Multi-view Transfer Learning (Multi-T).
Spam Detection
In spam detection, "content" features are often unstable as spammers actively attempt to bypass filters, whereas "sender" features remain more consistent. FEAD's ability to re-weight these groups is particularly effective in the low false-positive region, which is critical for avoiding the misclassification of legitimate emails.
In the "time-shifted" scenario, FEAD achieved a normalized Area Under the Curve (AUC) of 0.677, significantly higher than the 0.592 achieved by standard Logistic Regression.
Conclusion and Future Work
FEAD provides a robust framework for domain adaptation by exploiting the varying stability of feature groups. While the current implementation relies on manual or simple heuristic grouping and requires a small amount of labeled target data, it offers a significant performance boost over traditional methods. Future research will focus on automating the discovery of optimal feature groupings and extending the method to unsupervised domain adaptation scenarios.
Summary updated