Augmenting Limited and Biased RCTs through Pseudo-Sample Matching-Based Observational Data Fusion Method

Kairong Han, Weidong Huang, Taiyang Zhou, Peng Zhen, Kun Kuang

This paper, a collaboration between Zhejiang University and Didi Chuxing, tackles a common headache in industrial machine learning: what do you do when your "gold standard" data is too small and surprisingly messy?

TL;DR

The authors introduce a Pseudo-Sample Matching method to combine small, biased Randomized Controlled Trials (RCTs) with massive-scale observational data. By generating "ideal" pseudo-samples and matching them to similar real-world observations, they expanded their training sets and achieved a 0.41% profit increase in a week-long online deployment—a massive gain in the high-volume ride-hailing industry.

What Problem Does This Paper Address?

In the world of ride-hailing (like Uber or DiDi), companies use uplift models to decide who gets a discount. The goal is to find "Persuadables"—users who only book a ride because of the discount—while avoiding "Sure Things" (who book anyway) and "Sleeping Dogs" (who might be annoyed by the notification).

To train these models, companies run RCTs (A/B tests). However, RCTs are expensive. In this paper's context, RCT data accounted for only 0.65% of total traffic. Furthermore, real-world RCTs are rarely perfect; they suffer from "link interference" (drivers and passengers interacting in ways that break randomness) and selection bias due to complex engineering constraints. This leads to models that don't generalize well to the other 99.35% of the population.

Methods

The authors propose a data fusion method called Pseudo-Sample Matching. Instead of treating the RCT as the infallible truth, they use it as a seed to "fish" for better data in the vast ocean of observational records.

  1. Bias Correction: They calculate the feature mean of the entire RCT population versus specific treatment groups to identify "bias vectors."
  2. Pseudo-Sample Construction: For every sample in the biased RCT, they create a "pseudo-sample." This is essentially a version of the original data point adjusted by the bias vector and a "fusion ratio" ($k$) to align it with the overall distribution.
  3. Matching via KNN: They use K-Nearest Neighbors (KNN) to find the $k$ most similar samples in the observational dataset that match these pseudo-samples.
  4. Bucketing for Scale: To handle high-dimensional data and industrial scale, they use a "bucketing" strategy. They first group data by critical features (like supply-demand ratios) and then perform matching within those buckets on a distributed computing cluster.

This approach is "empirical"—it doesn't rely on the strict, often unverifiable assumptions (like SUTVA) required by many theoretical causal inference models.

Datasets

  • Synthetic Data: Used for controlled simulations where the "ground truth" causal effect is known. It included 20-dimensional features with complex distributions (Poisson, Binomial, Bimodal).
  • Real-World Industrial Data: Provided by DiDi Chuxing.
  • Offline: Data from 200+ cities in China over one year, featuring 33 different discount types.
  • Online: A week-long experiment across 62 major cities in China.

Key Insights and Ideas

  • The "Gold Standard" is Tarnished: A major takeaway is that in complex industrial systems, RCTs are often biased. Relying solely on them can lead to "contradictory trends" where offline metrics look great, but online performance tanks.
  • Observational Data as a Corrective: Rather than just using observational data to increase volume, this method uses it to correct the distributional shift of the RCT.
  • The Divergence Warning: The authors highlight that if RCT quality is low, increasing the fusion ratio can actually make biased evaluation metrics (like Qini) look worse while the actual model performance on the ground truth improves. This underscores the danger of over-optimizing for offline metrics on low-quality test sets.

Key Results

  • Simulation: The proposed method consistently outperformed standard models like TARNet and Causal Forests across Qini coefficients and Mean Absolute Percentage Error (MAPE).
  • Online Impact: In a massive-scale A/B test, the model trained with fused data achieved:
  • +0.87% in Gross Merchandise Value (GMV).
  • +0.41% in overall profit.
  • Scale: While 0.41% sounds small, the authors note this represents a "considerable gain" when scaled to scenarios involving hundreds of millions in revenue.

Limitations and Future Work

  • Feature Selection: The method relies on calculating feature means and distances. If the "important" features for matching aren't correctly identified by domain experts, the matching quality may suffer.
  • Computational Complexity: While bucketing and KD-trees help, KNN is still computationally expensive compared to simpler data augmentation.
  • Assumption of Overlap: The method assumes that the "ideal" samples needed to fix the RCT actually exist within the observational data pool. If the observational data is also missing those segments of the population, the fusion won't help.

The authors suggest that future work should focus on more automated ways to improve RCT data quality directly at the source in industrial settings.

Summary updated

Abstract

In the online ride-hailing pricing context, companies often conduct randomized controlled trials (RCTs) and utilize uplift models to assess the effect of discounts on customer orders, which substantially influences competitive market outcomes. However, due to the high cost of RCTs, the proportion of trial data relative to observational data is small, which only accounts for 0.65\% of total traffic in our context, resulting in significant bias when generalizing to the broader user base. Additionally, the complexity of industrial processes reduces the quality of RCT data, which is often subject to heterogeneity from potential interference and selection bias, making it difficult to correct. Moreover, existing data fusion methods are challenging to implement effectively in complex industrial settings due to the high dimensionality of features and the strict assumptions that are hard to verify with real-world data. To address these issues, we propose an empirical data fusion method called pseudo-sample matching. By generating pseudo-samples from biased, low-quality RCT data and matching them with the most similar samples from large-scale observational data, the method expands the RCT dataset while mitigating its heterogeneity. We validated the method through simulation experiments, conducted offline and online tests using real-world data. In a week-long online experiment, we achieved a 0.41\% improvement in profit, which is a considerable gain when scaled to industrial scenarios with hundreds of millions in revenue. In addition, we discuss the harm to model training, offline evaluation, and online economic benefits when the RCT data quality is not high, and emphasize the importance of improving RCT data quality in industrial scenarios. Further details of the simulation experiments can be found in the GitHub repository https://github.com/Kairong-Han/Pseudo-Matching.
BibTeX
@inproceedings{han2025augmenting,
  author = {Kairong Han and Weidong Huang and Taiyang Zhou and Peng Zhen and Kun Kuang},
  title = {{Augmenting Limited and Biased RCTs through Pseudo-Sample Matching-Based Observational Data Fusion Method}},
  booktitle = {ACM International Conference on Information and Knowledge Management},
  year = {2025},
  doi = {10.1145/3746252.3761532},
}

Loading related content...

Figure