Abstract
In production machine learning, user feedback is a vital signal for model improvement. However, "no news" is not necessarily "good news." A user's failure to correct a model prediction may stem from agreement or simply from being too busy to respond. Traditional crowdsourcing algorithms often fail in these environments by assuming silence implies correctness, leading to a feedback loop that reinforces model errors. We propose SURF (Selective Use of useR Feedback), an algorithm that explicitly models user busyness using an Expectation-Maximization framework. By treating the production classifier as a peer and estimating a per-user busyness probability, SURF effectively filters out noisy non-responses. Experiments on simulated MNIST production data demonstrate that SURF significantly outperforms state-of-the-art methods, maintaining high accuracy even as user busyness and classifier error rates increase.
Introduction
The deployment of machine learning models into production environments creates a unique challenge: learning from end-user interactions. Ideally, users provide corrections when a model fails (e.g., mislabeling a transaction). However, real-world users are often overwhelmed or disengaged, leading to an ambiguity of non-response. If a system assumes every non-response is an implicit "correct" label, it risks retraining on "silent" errors, causing performance to spiral downward. This paper addresses the need to distinguish between a "diligent" user who agrees with a label and a "busy" user whose silence should be disregarded.
Methodology: The SURF Algorithm
SURF extends the classic Dawid-Skene algorithm to account for the specific dynamics of production feedback loops.
The Busyness Parameter
The core innovation of SURF is the introduction of a busyness probability ($\epsilon$) for each user. Unlike traditional models that assume user independence, SURF recognizes that a busy user's lack of feedback is correlated with the classifier's output.
Expectation-Maximization Framework
SURF utilizes an EM loop to iteratively estimate the "ground truth" labels, the confusion matrix (reliability) of each user, and the busyness parameter for each user. By evaluating the model's own uncertainty, SURF can identify when a user's silence is suspicious. If the classifier is uncertain and the user provides no feedback, SURF is more likely to categorize that user as "busy" rather than "in agreement," thereby preventing the propagation of errors.
The Classifier as a Peer
A key feature of SURF is treating the production classifier itself as another "user" in the pool. This allows the system to learn the classifier's specific noise profile and balance its predictions against human feedback without inherent bias.
Experimental Setup
The researchers evaluated SURF using a simulated production environment based on the MNIST dataset:
- Ground Truth: Actual MNIST handwritten digit labels.
- Classifier Labels: Generated by a simulated agent with a fixed error probability ($p_c$).
- User Feedback: Generated by a pool of simulated "busy" agents with "pairwise-flipper" confusion matrices and a busyness rate ($\epsilon$).
Results and Discussion
SURF was compared against several state-of-the-art crowdsourcing algorithms, including Dawid-Skene, EBCC, and GLAD.
Resilience to Busyness
As shown in the figures, SURF remains remarkably stable as the busyness probability ($\epsilon$) increases. While traditional algorithms see a sharp decline in accuracy when $\epsilon > 0.5$, SURF maintains high performance by correctly identifying and discounting non-responses from busy users.
Handling Noisy Classifiers
SURF is particularly effective when the initial production classifier is poor ($p_c = 0.75$). In these scenarios, competitors often drop below 50% accuracy as busyness increases, while SURF maintains near-perfect accuracy in recovering ground truth. This resilience makes it ideal for cold-start problems or highly volatile production environments.
Conclusion
SURF provides a robust framework for learning from human-in-the-loop feedback by explicitly modeling the reality of user busyness. By distinguishing between intentional silence and lack of engagement, it prevents the performance degradation common in traditional production feedback loops. Future work will explore heterogeneous user pools and validation on real-world enterprise datasets.
Summary updated