Abstract
Weakly supervised learning aims at coping with scarce labeled data. Previous weakly supervised studies typically assume that there is only one kind of weak supervision in data. In many applications, however, raw data usually contains more than one kind of weak supervision at the same time. For example, in user experience enhancement from Didi, one of the largest online ride-sharing platforms, the ride comment data contains severe label noise (due to the subjective factors of passengers) and severe label distribution bias (due to the sampling bias). We call such a problem as ‘compound weakly supervised learning’. In this paper, we propose the CWSL method to address this problem based on Didi ride-sharing comment data. Specifically, an instance reweighting strategy is employed to cope with severe label noise in comment data, where the weights for harmful noisy instances are small. Robust criteria like AUC rather than accuracy and the validation performance are optimized for the correction of biased data label. Alternating optimization and stochastic gradient methods accelerate the optimization on large-scale data. Experiments on Didi ride-sharing comment data clearly validate the effectiveness. We hope this work may shed some light on applying weakly supervised learning to complex real situations.
Introduction
In traditional supervised learning, the assumption is that models are trained on massive, perfectly labeled datasets. However, in high-volume industries like ride-sharing, data is often "weak." Most research focuses on a single type of weakness—either label noise (incorrect labels) or distribution bias (training data mismatching the real-world distribution).
This paper introduces the concept of Compound Weakly Supervised Learning (CWSL) to address scenarios where both weaknesses coexist. In Didi's user comment system, for instance, passengers may leave negative ratings due to unrelated moods (label noise), and negative comments are naturally rare (distribution bias). Existing methods that solve one problem often fail when the other is present. Label noise cleaners ignore distribution shifts, while bias-correction methods assume the labels they have are perfectly accurate.
The CWSL Framework
The proposed CWSL method utilizes a bi-level optimization strategy to simultaneously clean data and train the model. The core idea is Instance Reweighting: instead of treating every training example equally, the model assigns a weight ($w$) to each instance. Noisy or harmful instances are assigned smaller weights to minimize their influence on the final model.
The optimization is nested in two levels:
- Lower Level: The model learns parameters ($\theta$) by minimizing a weighted loss on the large, noisy training set.
- Upper Level: The weights ($w$) are optimized to maximize performance on a small, clean, expert-verified validation set.
To handle extreme class imbalances (where negative comments are rare), the model optimizes for Area Under the ROC Curve (AUC) rather than simple accuracy. This ensures the model remains sensitive to the rare negative comments critical for service improvement.
Experimental Results
The CWSL method was evaluated on a real-world Didi dataset consisting of 600,000 noisy training instances and 15,000 expert-verified validation/test instances. The results demonstrate that CWSL significantly outperforms standard baselines and state-of-the-art weak supervision methods.
Key performance highlights include:
- Superior AUC: CWSL achieved an AUC of 91.24%, a substantial improvement over the best baseline (LTR at 85.45%) and standard XGBoost (79.55%).
- Robustness: As shown in the performance plots, the proposal remains consistently superior even as the ratio of positive to negative samples in the validation set changes.
- Efficiency: Despite the theoretical complexity of bi-level optimization, the authors' alternating gradient descent approach proved computationally feasible for large-scale industrial datasets, often running faster than simpler label-noise methods.
Conclusion
CWSL demonstrates that a small "gold set" of expert-verified data can effectively supervise the reweighting of hundreds of thousands of noisy samples. By treating noise and bias as an integrated optimization problem and utilizing AUC as a training objective, the model achieves state-of-the-art performance in identifying critical negative user feedback. Future work will explore extending this framework to multi-class problems and real-time dynamic adaptation.
Summary updated