TL;DR
Inspired by how humans ignore irrelevant or confusing information to focus better, Learning by Ignoring (LBI) is a novel machine learning framework that automatically identifies and excludes harmful pretraining data. By formulating data selection as a three-level optimization problem, LBI filters out out-of-domain examples, significantly boosting downstream domain adaptation performance.
What Problem Does This Paper Address?
Pretraining on large source datasets followed by finetuning on a target dataset is a standard recipe in deep learning. However, source datasets often contain examples with a significant "domain shift" from the target data.
Standard domain adaptation (DA) methods try to force the model to align all source data with the target domain. This can backfire if certain source examples are completely irrelevant, overly confusing, or noisy, leading to biased feature representations that hurt target task performance. The authors address this by asking: Can a model automatically learn what to ignore during pretraining?
Methods
The authors introduce Learning by Ignoring (LBI), which assigns a continuous "ignoring variable" $a_i \in [0, 1]$ to each pretraining example. If $a_i$ is close to 0, the example's loss is zeroed out, effectively removing it from training.
To learn these variables, LBI uses a three-level optimization framework consisting of three interconnected stages:
- Pretraining: An encoder is trained on the source data by minimizing a loss weighted by the ignoring variables ($A$).
- Finetuning: A second encoder is trained on the target data, regularized to stay close to the optimized pretrained encoder from Stage 1.
- Validation (Meta-update): The ignoring variables ($A$) are updated by minimizing the validation loss of the finetuned model on the target task.
Because solving this nested optimization exactly is computationally intractable, the authors develop an efficient gradient-based approximation algorithm using one-step gradient descent updates, drawing inspiration from meta-learning techniques like DARTS.
Datasets
The framework was evaluated on three standard domain adaptation benchmarks:
- Office31: 4,110 images across 31 categories from 3 domains (Amazon website, Web camera, and digital SLR camera).
- ImageCLEF: 1,800 images across 12 classes from 3 domains (Caltech-256, ImageNet, and Pascal VOC).
- OfficeHome: A more challenging dataset containing 15,500 images across 65 categories from 4 distinct domains (Art, Clipart, Product, and Real-World).
Key Insights and Ideas
- Selective Adaptation Beats Blanket Adaptation: Traditional domain adaptation assumes all source data is useful if properly aligned. LBI proves that actively discarding harmful out-of-domain data yields cleaner, more robust representations.
- Finetuning with Source Data can be a Double-Edged Sword: The authors found that blindly mixing source data into the finetuning stage hurts performance when domain shifts are large (especially in the OfficeHome dataset). However, applying the ignoring mechanism during finetuning flips this effect, turning a hazard into a helpful data boost.
- Implicit Data Distillation: By optimizing the pretraining weights based directly on target validation performance, the model implicitly distills the source dataset down to the core examples that genuinely benefit the target task.
Key Results
LBI was compared against popular domain adaptation methods (DAN, DANN, CDAN, MME) across unsupervised, semi-supervised, and supervised settings:
- Office31: LBI achieved the highest average accuracy of 94.79%, outperforming the best baseline (DAN-3 at 94.40%).
- ImageCLEF: LBI reached 84.50% average accuracy, a notable improvement over the top baseline (CDAN-3 at 82.89%).
- OfficeHome: LBI achieved an average accuracy of 73.19%, outperforming all standard DA baselines (the best being DAN-3 at 72.90%).
Ablation studies confirmed that ignoring bad data during both pretraining and finetuning provided the best overall results.
Limitations and Future Work
Not explicitly discussed in the paper. However, based on the methodology, potential limitations include the computational overhead of calculating higher-order gradients (Hessian-vector products) during the meta-update step, even with the one-step approximation. Future work could explore extending LBI to more complex tasks like object detection or natural language processing where domain shifts are highly non-linear.
Summary updated