Learning by Ignoring, with Application to Domain Adaptation

Xingchen Zhao, Xuehai He, Pengtao Xie

TL;DR

Inspired by how humans ignore irrelevant or confusing information to focus better, Learning by Ignoring (LBI) is a novel machine learning framework that automatically identifies and excludes harmful pretraining data. By formulating data selection as a three-level optimization problem, LBI filters out out-of-domain examples, significantly boosting downstream domain adaptation performance.

What Problem Does This Paper Address?

Pretraining on large source datasets followed by finetuning on a target dataset is a standard recipe in deep learning. However, source datasets often contain examples with a significant "domain shift" from the target data.

Standard domain adaptation (DA) methods try to force the model to align all source data with the target domain. This can backfire if certain source examples are completely irrelevant, overly confusing, or noisy, leading to biased feature representations that hurt target task performance. The authors address this by asking: Can a model automatically learn what to ignore during pretraining?

Methods

The authors introduce Learning by Ignoring (LBI), which assigns a continuous "ignoring variable" $a_i \in [0, 1]$ to each pretraining example. If $a_i$ is close to 0, the example's loss is zeroed out, effectively removing it from training.

To learn these variables, LBI uses a three-level optimization framework consisting of three interconnected stages:

  1. Pretraining: An encoder is trained on the source data by minimizing a loss weighted by the ignoring variables ($A$).
  2. Finetuning: A second encoder is trained on the target data, regularized to stay close to the optimized pretrained encoder from Stage 1.
  3. Validation (Meta-update): The ignoring variables ($A$) are updated by minimizing the validation loss of the finetuned model on the target task.
Overview of the LBI framework loop

Because solving this nested optimization exactly is computationally intractable, the authors develop an efficient gradient-based approximation algorithm using one-step gradient descent updates, drawing inspiration from meta-learning techniques like DARTS.

Detailed three-stage optimization flow in LBI

Datasets

The framework was evaluated on three standard domain adaptation benchmarks:

  • Office31: 4,110 images across 31 categories from 3 domains (Amazon website, Web camera, and digital SLR camera).
  • ImageCLEF: 1,800 images across 12 classes from 3 domains (Caltech-256, ImageNet, and Pascal VOC).
  • OfficeHome: A more challenging dataset containing 15,500 images across 65 categories from 4 distinct domains (Art, Clipart, Product, and Real-World).

Key Insights and Ideas

  • Selective Adaptation Beats Blanket Adaptation: Traditional domain adaptation assumes all source data is useful if properly aligned. LBI proves that actively discarding harmful out-of-domain data yields cleaner, more robust representations.
  • Finetuning with Source Data can be a Double-Edged Sword: The authors found that blindly mixing source data into the finetuning stage hurts performance when domain shifts are large (especially in the OfficeHome dataset). However, applying the ignoring mechanism during finetuning flips this effect, turning a hazard into a helpful data boost.
  • Implicit Data Distillation: By optimizing the pretraining weights based directly on target validation performance, the model implicitly distills the source dataset down to the core examples that genuinely benefit the target task.

Key Results

LBI was compared against popular domain adaptation methods (DAN, DANN, CDAN, MME) across unsupervised, semi-supervised, and supervised settings:

  • Office31: LBI achieved the highest average accuracy of 94.79%, outperforming the best baseline (DAN-3 at 94.40%).
  • ImageCLEF: LBI reached 84.50% average accuracy, a notable improvement over the top baseline (CDAN-3 at 82.89%).
  • OfficeHome: LBI achieved an average accuracy of 73.19%, outperforming all standard DA baselines (the best being DAN-3 at 72.90%).

Ablation studies confirmed that ignoring bad data during both pretraining and finetuning provided the best overall results.

Limitations and Future Work

Not explicitly discussed in the paper. However, based on the methodology, potential limitations include the computational overhead of calculating higher-order gradients (Hessian-vector products) during the meta-update step, even with the one-step approximation. Future work could explore extending LBI to more complex tasks like object detection or natural language processing where domain shifts are highly non-linear.

Summary updated

Abstract

Learning by ignoring, which identifies less important things and excludes them from the learning process, is broadly practiced in human learning and has shown ubiquitous effectiveness. There has been psychological studies showing that learning to ignore certain things is a powerful tool for helping people focus. In this paper, we explore whether this useful human learning methodology can be borrowed to improve machine learning. We propose a novel machine learning framework referred to as learning by ignoring (LBI). Our framework automatically identifies pretraining data examples that have large domain shift from the target distribution by learning an ignoring variable for each example and excludes them from the pretraining process. We formulate LBI as a three-level optimization framework where three learning stages are involved: pretraining by minimizing the losses weighed by ignoring variables; finetuning; updating the ignoring variables by minimizing the validation loss. A gradient-based algorithm is developed to efficiently solve the three-level optimization problem in LBI. Experiments on various datasets demonstrate the effectiveness of our framework.
BibTeX
@misc{zhao2020learning,
  author = {Xingchen Zhao and Xuehai He and Pengtao Xie},
  title = {{Learning by Ignoring, with Application to Domain Adaptation}},
  year = {2020},
  eprint = {2012.14288},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.14288},
}
Figure