This paper, a collaboration between Zhejiang University and Didi Chuxing, tackles a common headache in industrial machine learning: what do you do when your "gold standard" data is too small and surprisingly messy?
TL;DR
The authors introduce a Pseudo-Sample Matching method to combine small, biased Randomized Controlled Trials (RCTs) with massive-scale observational data. By generating "ideal" pseudo-samples and matching them to similar real-world observations, they expanded their training sets and achieved a 0.41% profit increase in a week-long online deployment—a massive gain in the high-volume ride-hailing industry.
What Problem Does This Paper Address?
In the world of ride-hailing (like Uber or DiDi), companies use uplift models to decide who gets a discount. The goal is to find "Persuadables"—users who only book a ride because of the discount—while avoiding "Sure Things" (who book anyway) and "Sleeping Dogs" (who might be annoyed by the notification).
To train these models, companies run RCTs (A/B tests). However, RCTs are expensive. In this paper's context, RCT data accounted for only 0.65% of total traffic. Furthermore, real-world RCTs are rarely perfect; they suffer from "link interference" (drivers and passengers interacting in ways that break randomness) and selection bias due to complex engineering constraints. This leads to models that don't generalize well to the other 99.35% of the population.
Methods
The authors propose a data fusion method called Pseudo-Sample Matching. Instead of treating the RCT as the infallible truth, they use it as a seed to "fish" for better data in the vast ocean of observational records.
- Bias Correction: They calculate the feature mean of the entire RCT population versus specific treatment groups to identify "bias vectors."
- Pseudo-Sample Construction: For every sample in the biased RCT, they create a "pseudo-sample." This is essentially a version of the original data point adjusted by the bias vector and a "fusion ratio" ($k$) to align it with the overall distribution.
- Matching via KNN: They use K-Nearest Neighbors (KNN) to find the $k$ most similar samples in the observational dataset that match these pseudo-samples.
- Bucketing for Scale: To handle high-dimensional data and industrial scale, they use a "bucketing" strategy. They first group data by critical features (like supply-demand ratios) and then perform matching within those buckets on a distributed computing cluster.
This approach is "empirical"—it doesn't rely on the strict, often unverifiable assumptions (like SUTVA) required by many theoretical causal inference models.
Datasets
- Synthetic Data: Used for controlled simulations where the "ground truth" causal effect is known. It included 20-dimensional features with complex distributions (Poisson, Binomial, Bimodal).
- Real-World Industrial Data: Provided by DiDi Chuxing.
- Offline: Data from 200+ cities in China over one year, featuring 33 different discount types.
- Online: A week-long experiment across 62 major cities in China.
Key Insights and Ideas
- The "Gold Standard" is Tarnished: A major takeaway is that in complex industrial systems, RCTs are often biased. Relying solely on them can lead to "contradictory trends" where offline metrics look great, but online performance tanks.
- Observational Data as a Corrective: Rather than just using observational data to increase volume, this method uses it to correct the distributional shift of the RCT.
- The Divergence Warning: The authors highlight that if RCT quality is low, increasing the fusion ratio can actually make biased evaluation metrics (like Qini) look worse while the actual model performance on the ground truth improves. This underscores the danger of over-optimizing for offline metrics on low-quality test sets.
Key Results
- Simulation: The proposed method consistently outperformed standard models like TARNet and Causal Forests across Qini coefficients and Mean Absolute Percentage Error (MAPE).
- Online Impact: In a massive-scale A/B test, the model trained with fused data achieved:
- +0.87% in Gross Merchandise Value (GMV).
- +0.41% in overall profit.
- Scale: While 0.41% sounds small, the authors note this represents a "considerable gain" when scaled to scenarios involving hundreds of millions in revenue.
Limitations and Future Work
- Feature Selection: The method relies on calculating feature means and distances. If the "important" features for matching aren't correctly identified by domain experts, the matching quality may suffer.
- Computational Complexity: While bucketing and KD-trees help, KNN is still computationally expensive compared to simpler data augmentation.
- Assumption of Overlap: The method assumes that the "ideal" samples needed to fix the RCT actually exist within the observational data pool. If the observational data is also missing those segments of the population, the fusion won't help.
The authors suggest that future work should focus on more automated ways to improve RCT data quality directly at the source in industrial settings.
Summary updated