Abstract
Domain shift remains a critical challenge in machine learning, where models trained on out-of-domain (OOD) data often fail to generalize to specific target domains. This paper introduces WIND (Weighting INstances Differentially), a model-agnostic framework that leverages meta-learning to automatically learn instance weights for training samples. By formulating instance weighting as a bi-level optimization problem, WIND identifies beneficial OOD samples based on their actual contribution to target task performance. Experimental results across sentiment classification, neural machine translation, and relation extraction demonstrate that WIND consistently outperforms traditional hand-crafted weighting and adversarial methods.
Introduction
Machine learning models frequently suffer from domain shift, a phenomenon where a model trained on one context (e.g., kitchen appliance reviews) fails when applied to another (e.g., furniture reviews). This occurs because the semantic importance of features can change across domains. While large-scale labeled out-of-domain (OOD) data is often available, effectively integrating it with limited in-domain (ID) data is difficult. Traditional instance weighting methods rely on manual similarity metrics that often fail to generalize across different tasks.
Methodology
WIND treats domain adaptation as a bi-level optimization problem inspired by Meta-Learning techniques like MAML. Instead of using a fixed formula to decide a sample's importance, WIND learns weights through a three-step iterative process:
- Pseudo Update: The model performs a "practice" training step on a mix of OOD and ID data using current weights.
- Instance Weight Update: The model evaluates its practice performance on a separate query set (a small, clean batch of in-domain data). It then updates the instance weights based on whether those samples actually improved performance on the query set.
- Final Update: The model performs the actual parameter update using the newly optimized weights.
To ensure computational efficiency, the authors employ a second-order derivation approximation (similar to DARTS), allowing the estimation of complex gradients without the prohibitive cost of calculating a full Hessian matrix.
Experimental Evaluation
The authors evaluated WIND across three distinct NLP tasks to prove its model-agnostic nature:
- Sentiment Classification: Using the Amazon Review Dataset.
- Neural Machine Translation (NMT): Using IWSLT 2016 (TED talks) as ID data and WMT 2014 as OOD data.
- Relation Extraction: Using the ACE 2005 dataset, adapting from Newswire/Broadcast News to Broadcast Conversation.
Key Results and Discussion
WIND consistently outperformed strong baselines, including Domain Adversarial Neural Networks (DANN). In sentiment classification, WIND achieved an average accuracy of 92.46%, maintaining a clear lead even when the in-domain training set was reduced to just 100 samples.
In Neural Machine Translation, WIND improved BLEU scores significantly. On the tst2012 and tst2013 sets, it achieved gains of 1.17 and 1.08 points respectively over standard joint training (In+Out).
For Relation Extraction, the framework reached 90.54% accuracy, a substantial improvement over the 68.05% achieved by training solely on in-domain data. These results highlight that meta-learning acts as an effective filter, identifying OOD sentences that truly benefit the model's target task performance rather than relying on simple surface-level similarity.
Conclusion
WIND provides a robust, model-agnostic approach to domain adaptation by making instance weighting differentiable. While the method introduces some computational overhead and depends on a high-quality query set, its ability to discover complex relationships between OOD data and target performance represents a significant advancement over static weighting schemes. Future work may explore more sophisticated knowledge transfer methods beyond instance weighting.
Summary updated