Abstract
Transfer learning often suffers when source data contains examples that are irrelevant or detrimental to the target task due to domain shift. We present "Learning by Ignoring" (LBI), a framework that treats data selection as a three-level optimization problem. LBI learns to assign weights to source examples, effectively "ignoring" those that do not contribute to target task performance. Evaluated on major domain adaptation benchmarks, LBI consistently outperforms state-of-the-art methods by focusing on the most relevant data subsets.
Introduction
In deep learning, the prevailing wisdom is that more data leads to better models. However, not all data is created equal. In transfer learning, where a model is pretrained on a large source dataset and finetuned on a specific target task, many source examples may exhibit significant domain shift. These irrelevant or noisy examples can actually degrade performance on the target task.
While humans naturally filter out distractions to focus on relevant information, machine learning models typically attempt to learn from every example provided. Researchers from UC San Diego and the University of Pittsburgh propose the Learning by Ignoring (LBI) framework to bridge this gap. LBI enables models to automatically identify and exclude detrimental source data, curating a more effective training set for the target domain.
The LBI Framework
The core of LBI is a three-level optimization problem designed to synchronize data selection with model training. Instead of a standard training loop, LBI operates through three interconnected stages:
- Weighted Pretraining: The model trains an encoder on the source dataset. Each source example is assigned an "ignoring variable" ($a \in [0, 1]$). If $a$ is near zero, the example’s contribution to the loss is nullified.
- Finetuning: A target encoder is trained on the target dataset, using a regularization term to maintain proximity to the weights of the pretrained encoder.
- Meta-Update: The "ignoring variables" are updated by minimizing the loss on a validation set of the target task.
To handle the computational complexity of three-level optimization, the authors utilize a gradient-based approximation algorithm. This allows the model to learn which source images to ignore while simultaneously training the encoder weights.
Experimental Results
LBI was evaluated on three standard domain adaptation benchmarks: Office31, ImageCLEF, and OfficeHome. The framework consistently outperformed state-of-the-art baselines, including DAN, DANN, and CDAN.
- Office31: Achieved an average accuracy of 94.79%, surpassing the best baseline (DAN-3) at 94.40%.
- ImageCLEF: Reached 84.50% average accuracy, a significant jump over the 82.89% achieved by CDAN-3.
- OfficeHome: On this more complex dataset, LBI hit 73.19%, outperforming the 72.90% of DAN-3.
A key insight from these results is the Power of "No". Traditional methods focus on aligning source and target features, but LBI demonstrates that simply excluding dissimilar source examples can be more effective. Ablation studies confirmed that the ignoring mechanism is vital; in some cases, using the full source dataset for finetuning actually decreased accuracy compared to using no source data at all, underscoring the risks of domain shift.
Conclusion and Future Work
The "Learning by Ignoring" framework represents a shift toward more discerning machine learning. By treating data selection as a differentiable optimization task guided by validation performance, LBI achieves superior transfer learning results.
While the current implementation introduces some computational overhead due to the multi-level optimization, it opens new avenues for data curation in AI. Future research may extend LBI to other domains such as Natural Language Processing or integrate it with self-supervised learning paradigms to further enhance model robustness against noisy data.
Summary updated