Learning from Samples of Variable Quality

Mostafa Dehghani, Arash Mehrjou, Stephan Gouws, Jaap Kamps, Bernhard SchölkopfUvA-DARE 2019

Abstract

Training labels are expensive to obtain and may be of varying quality, as some may be from trusted expert labelers while others might be from heuristics or other sources of weak supervision such as crowd-sourcing. This creates a fundamental quality-versus-quantity trade-off in the learning process. Do we learn from the small amount of high-quality data or the potentially large amount of weakly-labeled data? We argue that if the learner could somehow know and take the label-quality into account, we could get the best of both worlds. To this end, we introduce “fidelity-weighted learning” (FWL), a semi-supervised student-teacher approach for training deep neural networks using weakly-labeled data. FWL modulates the parameter updates to a student network, trained on the task we care about on a per-sample basis according to the posterior confidence of its label-quality estimated by a teacher, who has access to limited samples with high-quality labels.

Introduction

In modern deep learning, the performance of models is often bottlenecked by the availability of high-quality labeled data. Researchers typically face a choice between two suboptimal paths: the "Strong Path," utilizing a small set of accurate expert labels that risk overfitting, or the "Weak Path," utilizing massive but noisy datasets generated via heuristics or crowd-sourcing. Standard techniques like pre-training on weak data followed by fine-tuning on strong data often fail to account for the inherent variance in label reliability. Fidelity-Weighted Learning (FWL) is proposed to resolve this dilemma by enabling the model to "know" which weak labels are likely erroneous and down-weighting them during training.

Methodology: Fidelity-Weighted Learning

FWL implements a three-step semi-supervised framework designed to leverage the strengths of both large-scale weak data and small-scale expert data.

  1. Student Pre-training: A deep neural network (the student) is first trained on a large, weakly-labeled dataset $\mathcal{D}_w$. This phase focuses on learning robust data representations (embeddings) $\psi(x_i)$.
  2. Teacher Training: A Bayesian model, specifically a Gaussian Process (GP), acts as the teacher. It is trained on a small set of high-quality labels $\mathcal{D}_s$. Crucially, the teacher operates on the student's learned representations $\psi^*(x_j)$, allowing it to estimate label quality through the student's "lens."
  3. Fidelity-Weighted Fine-tuning: The student is fine-tuned using soft labels generated by the teacher. The core innovation is the modulation of the learning rate $\eta$ on a per-sample basis. If the teacher expresses high uncertainty (variance) regarding a sample, the student's parameter update for that sample is minimized.
The three steps of Fidelity-Weighted Learning: (a) Student pre-training on weak labels, (b) Teacher training on strong labels using learned representations, and (c) Student fine-tuning with fidelity-weighted updates.

The learning rate modulation is governed by the function $\eta_2(x_t) = \exp[-\beta\Sigma(x_t)]$, where $\Sigma$ represents the teacher's posterior uncertainty. The hyperparameter $\beta$ allows for tuning the bias-variance trade-off between the teacher's guidance and the student's initial knowledge.

Application to Document Ranking

The authors evaluated FWL on the task of Document Ranking using the Robust04 and ClueWeb (Category B) benchmarks. For these experiments, the student network utilized a specific neural ranking architecture that processes query-document pairs through embedding and compositionality layers before ranking.

The neural ranking architecture used as the student network, featuring embedding layers and a compositionality module to process query-document pairs.

Weak labels were generated using the BM25 algorithm, while strong labels consisted of official TREC query judgments. This setup allowed the researchers to test how well FWL could refine a model initially trained on simple heuristic-based rankings using a small set of expert relevance judgments.

Key Results and Insights

FWL demonstrated significant performance gains over traditional supervised learning and standard fine-tuning baselines. On the Robust04 dataset, FWL achieved a Mean Average Precision (MAP) of 0.3124, a substantial improvement over the 0.2810 achieved by standard fine-tuning ($NN_{W \to S}$) and the 0.1790 from training solely on strong labels ($NN_S$).

Key findings include:

  • Robustness: FWL consistently improved results even when the initial weak labels were of lower quality.
  • Efficiency: By filtering out unreliable samples via the learning rate, the framework effectively speeds up the convergence toward high-performing parameters.
  • Representational Synergy: The success of the teacher is tied to the quality of the student's internal representations, highlighting a symbiotic relationship between the two components.

Conclusion

Fidelity-Weighted Learning provides a principled Bayesian approach to handling training data of varying quality. By treating the learning rate as a per-sample fidelity filter, FWL allows deep learning models to benefit from the scale of weak supervision without being misled by its noise. Future work may explore scaling this approach to other domains, such as computer vision, where automated or crowd-sourced labels are prevalent but imperfect.

Summary updated

Abstract

Training labels are expensive to obtain and may be of varying quality, as some may be from trusted expert labelers while others might be from heuristics or other sources of weak supervision such as crowd-sourcing. This creates a fundamental quality-versus-quantity trade-off in the learning process. Do we learn from the small amount of high-quality data or the potentially large amount of weakly-labeled data? We argue that if the learner could somehow know and take the label-quality into account, we could get the best of both worlds. To this end, we introduce “fidelity-weighted learning” (FWL), a semi-supervised student-teacher approach for training deep neural networks using weakly-labeled data. FWL modulates the parameter updates to a student network, trained on the task we care about on a per-sample basis according to the posterior confidence of its label-quality estimated by a teacher, who has access to limited samples with high-quality labels.
BibTeX
@inproceedings{dehghani2019learning,
  author = {Mostafa Dehghani and Arash Mehrjou and Stephan Gouws and Jaap Kamps and Bernhard Schölkopf},
  title = {{Learning from Samples of Variable Quality}},
  booktitle = {UvA-DARE},
  year = {2019},
}
Figure