Some people aren't worth listening to: periodically retraining classifiers with feedback from a team of end users

Joshua Lockhart, Samuel Assefa, Tucker Balch, Manuela VelosoarXiv 2020 · arXiv

Abstract

In real-world machine learning deployments, classifiers often require periodic retraining using feedback from end-users to combat data drift. However, this feedback loop is vulnerable to "noisy" or "confused" users whose labels may be incorrect. This paper, by researchers at JP Morgan AI Research, proposes PRUNE (Periodically Retraining Using Native Entities), a framework that maintains individual "trust scores" for users. By filtering out feedback from unreliable agents, PRUNE prevents model degradation and significantly outperforms naive retraining strategies, matching the performance of an ideal system with perfect knowledge of user reliability.

Introduction

Document classifiers in production environments, such as financial institutions, inevitably face data drift or novel inputs. A common solution is the "feedback-retrain loop," where analysts provide corrections for misclassified items. Traditional crowdsourcing relies on redundant labeling (e.g., majority voting), but business workflows typically assign a document to a single person. This creates a risk: if the user is unreliable, their feedback can actively degrade the model. The authors categorize unreliable users into two types: noisy agents, who provide random labels, and confused agents, who consistently misinterpret specific categories.

The PRUNE Framework

The authors introduce the PRUNE wrapper to manage the retraining process. The core innovation is the Trust Score ($T$), which quantifies an agent's reliability based on their agreement with the current classifier:

$$T(i) = \frac{\text{Agreement count between classifier and agent } A_i}{\text{Total labels provided by agent } A_i}$$

Two filtering strategies are evaluated:

  1. MEANPRUNE: Accepts feedback only from agents with trust scores above the group average.
  2. THRESHOLDPRUNE: Accepts feedback only if the trust score exceeds a fixed threshold (e.g., 0.3).

To avoid the "cold start" problem—where an initially weak classifier might unfairly distrust a competent user—the system employs a "burn-in" period. During this phase, all feedback is accepted to allow the classifier to stabilize before it begins evaluating user reliability.

Experimental Results

The framework was tested on the 20 Newsgroups (20N) and Reuters-21578 (reu) datasets. The authors compared PRUNE against TRUSTINGNB (accepts all feedback) and DISCERNINGNB (an oracle with perfect knowledge of agent reliability).

Resilience to Confused Agents

Confused agents, who provide "structured misinformation," were found to be significantly more damaging than noisy agents. Naive retraining (TRUSTINGNB) often led to immediate performance drops when faced with confused feedback. In contrast, PRUNE algorithms successfully identified and muted these agents.

Performance on the Reuters dataset with confused agents. The PRUNE method (solid green) maintains high accuracy and matches the oracle benchmark (dash-dot orange), while the naive approach (dashed blue) degrades.

On the Reuters dataset, the filtering was so effective that it reached the same performance level as the perfect-knowledge benchmark by the later stages of the trial.

Resilience to Noisy Agents

While noisy feedback acts as statistical jitter, it can still hinder learning. In scenarios with a high ratio of noisy agents (e.g., 4 out of 5), PRUNE consistently outperformed the "trust everyone" baseline.

Performance on the 20 Newsgroups dataset. PRUNE methods effectively filter out noise to maintain steady improvement in classifier accuracy over multiple retraining stages.

Discussion and Limitations

A key insight is the "Misplaced Trust" trap: if a classifier makes an error and a reliable agent corrects it, the disagreement lowers the agent's trust score. The "burn-in" period is critical to mitigate this. However, some trials still exhibited bimodal results, where the model either successfully identified reliable agents or failed completely.

Future work could extend this model to handle dynamic user reliability (e.g., learning or fatigue) and larger agent pools. Additionally, trust scores could serve as a diagnostic tool to identify users in need of further training.

Conclusion

PRUNE offers a robust solution for maintaining classifiers in human-in-the-loop systems. By treating users as persistent agents with measurable reliability, the system can automatically clean its training data stream, ensuring that "some people aren't worth listening to" doesn't lead to model failure.

Summary updated

Abstract

Document classification is ubiquitous in a business setting, but often the end users of a classifier are engaged in an ongoing feedback-retrain loop with the team that maintain it. We consider this feedback-retrain loop from a multi-agent point of view, considering the end users as autonomous agents that provide feedback on the labelled data provided by the classifier. This allows us to examine the effect on the classifier's performance of unreliable end users who provide incorrect feedback. We demonstrate a classifier that can learn which users tend to be unreliable, filtering their feedback out of the loop, thus improving performance in subsequent iterations.
BibTeX
@misc{lockhart2020some,
  author = {Joshua Lockhart and Samuel Assefa and Tucker Balch and Manuela Veloso},
  title = {{Some people aren't worth listening to: periodically retraining classifiers with feedback from a team of end users}},
  year = {2020},
  eprint = {2004.13152},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2004.13152},
}
Figure