Abstract
Weakly supervised learning aims at coping with scarce labeled data. Previous weakly supervised studies typically assume that there is only one kind of weak supervision in data. In many applications, however, raw data usually contains more than one kind of weak supervision at the same time. For example, in user experience enhancement from Didi, one of the largest online ride-sharing platforms, the ride comment data contains severe label noise (due to the subjective factors of passengers) and severe label distribution bias (due to the sampling bias). We call such a problem as "compound weakly supervised learning". In this paper, we propose the CWSL method to address this problem based on Didi ride-sharing comment data. Specifically, an instance reweighting strategy is employed to cope with severe label noise in comment data, where the weights for harmful noisy instances are small. Robust criteria like AUC rather than accuracy and the validation performance are optimized for the correction of biased data label. Alternating optimization and stochastic gradient methods accelerate the optimization on large-scale data. Experiments on Didi ride-sharing comment data clearly validate the effectiveness. We hope this work may shed some light on applying weakly supervised learning to complex real situations.
Introduction
In traditional supervised learning, the ideal scenario involves a massive, perfectly labeled dataset. However, in real-world industrial applications, labeling is often expensive and prone to error. While Weakly Supervised Learning (WSL) was developed to handle imperfections like noisy, incomplete, or coarse labels, most existing research focuses on a single type of weakness at a time.
This paper addresses the challenge of Compound Weakly Supervised Learning (CWSL), where multiple weaknesses coexist. Using Didi’s ride-sharing platform as a case study, the authors identify two primary issues in passenger comment data: severe label noise due to subjective emotional responses and significant label distribution bias, where negative comments are rare (approximately 1:30) and their nature shifts over time. CWSL provides a unified framework to tackle both noise and bias simultaneously.
The CWSL Framework
The proposed CWSL method utilizes a bi-level optimization task to refine the learning process. The framework is designed to leverage a large, noisy training set alongside a small, expert-verified "gold standard" validation set.
As illustrated in the framework overview, the process involves:
- Lower-Level Optimization (Inner Loop): The model learns from the large training set ($x^t, y^t$). Instead of treating all instances equally, it applies an instance reweighting mechanism ($w$). The goal is to minimize the weighted empirical risk.
- Upper-Level Optimization (Outer Loop): This level tunes the weights ($w$) by evaluating the model's performance on a small, clean validation set ($x^v, y^v$). The weights are adjusted to maximize performance metrics on this high-quality data.
- Robust AUC Criteria: Given the extreme class imbalance in ride-sharing data, the authors optimize for the Area Under the Curve (AUC) rather than simple accuracy. To ensure differentiability for neural network training, a pairwise squared loss is used as a smooth surrogate for AUC.
Experimental Setup and Data
The effectiveness of CWSL was tested on two primary datasets:
- Synthetic Data: Using the UCI Breast Cancer dataset, the authors manually introduced label noise to verify if the algorithm could successfully identify and down-weight corrupted instances.
- Real-World Didi Data: A massive dataset of 600,000 training instances containing natural noise and bias. For evaluation, 15,000 instances from a later time period were verified by human experts to serve as clean validation and test sets.
Interpreting Instance Weights
A key advantage of the CWSL approach is the interpretability of the reweighting mechanism. The weights act as a diagnostic tool, effectively filtering the dataset by identifying which instances are likely noisy and which are clean.
As shown in the weight distribution histograms, the model progressively learns to distinguish between data types. By 100 epochs, the weights for noisy instances are pushed toward zero, while clean instances maintain high weights near 1.0. This demonstrates that a tiny, high-quality validation set can act as a "compass" to guide training on millions of noisy rows.
Key Results and Performance
CWSL was compared against standard baselines (XGBoost, DNN) and state-of-the-art weak supervision methods (Rank Pruning, GLC, LTR). The results highlight the superiority of the compound approach:
- AUC Gains: CWSL achieved an AUC of 91.24, significantly outperforming the best baseline (LTR at 85.45) and standard XGBoost (79.55).
- F1-Score Improvement: In the face of extreme class imbalance, CWSL reached an F1-score of 27.41%, nearly doubling the performance of several other methods that struggled to break 20%.
- Efficiency: Despite the complexity of bi-level optimization, the alternating optimization strategy keeps computational overhead manageable—only about 2x that of standard training—making it viable for industrial deployment.
Conclusion
CWSL demonstrates that addressing multiple data weaknesses in a unified framework is both possible and highly effective for real-world applications. While the method requires a small set of expert-verified data, the performance gains in noisy and imbalanced environments are substantial. Future work will explore making these models more dynamic to handle the shifting user behaviors inherent in ride-sharing environments.
Summary updated