Learning by Ignoring

Xingchen Zhao, Pengtao Xie

Abstract

Transfer learning often suffers when source data contains examples that are irrelevant or detrimental to the target task due to domain shift. We present "Learning by Ignoring" (LBI), a framework that treats data selection as a three-level optimization problem. LBI learns to assign weights to source examples, effectively "ignoring" those that do not contribute to target task performance. Evaluated on major domain adaptation benchmarks, LBI consistently outperforms state-of-the-art methods by focusing on the most relevant data subsets.

Introduction

In deep learning, the prevailing wisdom is that more data leads to better models. However, not all data is created equal. In transfer learning, where a model is pretrained on a large source dataset and finetuned on a specific target task, many source examples may exhibit significant domain shift. These irrelevant or noisy examples can actually degrade performance on the target task.

While humans naturally filter out distractions to focus on relevant information, machine learning models typically attempt to learn from every example provided. Researchers from UC San Diego and the University of Pittsburgh propose the Learning by Ignoring (LBI) framework to bridge this gap. LBI enables models to automatically identify and exclude detrimental source data, curating a more effective training set for the target domain.

Overview of the LBI framework showing the flow from pretraining to validation.

The LBI Framework

The core of LBI is a three-level optimization problem designed to synchronize data selection with model training. Instead of a standard training loop, LBI operates through three interconnected stages:

  1. Weighted Pretraining: The model trains an encoder on the source dataset. Each source example is assigned an "ignoring variable" ($a \in [0, 1]$). If $a$ is near zero, the example’s contribution to the loss is nullified.
  2. Finetuning: A target encoder is trained on the target dataset, using a regularization term to maintain proximity to the weights of the pretrained encoder.
  3. Meta-Update: The "ignoring variables" are updated by minimizing the loss on a validation set of the target task.
Detailed architecture of the three-level optimization process in LBI.

To handle the computational complexity of three-level optimization, the authors utilize a gradient-based approximation algorithm. This allows the model to learn which source images to ignore while simultaneously training the encoder weights.

Experimental Results

LBI was evaluated on three standard domain adaptation benchmarks: Office31, ImageCLEF, and OfficeHome. The framework consistently outperformed state-of-the-art baselines, including DAN, DANN, and CDAN.

  • Office31: Achieved an average accuracy of 94.79%, surpassing the best baseline (DAN-3) at 94.40%.
  • ImageCLEF: Reached 84.50% average accuracy, a significant jump over the 82.89% achieved by CDAN-3.
  • OfficeHome: On this more complex dataset, LBI hit 73.19%, outperforming the 72.90% of DAN-3.

A key insight from these results is the Power of "No". Traditional methods focus on aligning source and target features, but LBI demonstrates that simply excluding dissimilar source examples can be more effective. Ablation studies confirmed that the ignoring mechanism is vital; in some cases, using the full source dataset for finetuning actually decreased accuracy compared to using no source data at all, underscoring the risks of domain shift.

Conclusion and Future Work

The "Learning by Ignoring" framework represents a shift toward more discerning machine learning. By treating data selection as a differentiable optimization task guided by validation performance, LBI achieves superior transfer learning results.

While the current implementation introduces some computational overhead due to the multi-level optimization, it opens new avenues for data curation in AI. Future research may extend LBI to other domains such as Natural Language Processing or integrate it with self-supervised learning paradigms to further enhance model robustness against noisy data.

Summary updated

Abstract

Learning by ignoring, which identifies less important things and excludes them from the learning process, is broadly practiced in human learning and has shown ubiquitous effectiveness. There has been psychological studies showing that learning to ignore certain things is a powerful tool for helping people focus. In this paper, we explore whether this useful human learning methodology can be borrowed to improve machine learning. We propose a novel machine learning framework referred to as learning by ignoring (LBI). Our framework automatically identifies pretraining data examples that have large domain shift from the target distribution by learning an ignoring variable for each example and excludes them from the pretraining process. We formulate LBI as a three-level optimization framework where three learning stages are involved: pretraining by minimizing the losses weighed by ignoring variables; finetuning; updating the ignoring variables by minimizing the validation loss. A gradient-based algorithm is developed to efficiently solve the three-level optimization problem in LBI. Experiments on various datasets demonstrate the effectiveness of our framework.
BibTeX
@misc{zhao2020learning,
  author = {Xingchen Zhao and Pengtao Xie},
  title = {{Learning by Ignoring}},
  year = {2020},
  eprint = {2012.14288},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.14288},
}
Figure