Augmenting Limited and Biased RCTs through Pseudo-Sample Matching-Based Observational Data Fusion Method

Kairong Han, Weidong Huang, Taiyang Zhou, Peng Zhen, Kun Kuang

Abstract

In the online ride-hailing pricing context, companies often conduct randomized controlled trials (RCTs) and utilize uplift models to assess the effect of discounts on customer orders, which substantially influences competitive market outcomes. However, due to the high cost of RCTs, the proportion of trial data relative to observational data is small, which only accounts for 0.65% of total traffic in our context, resulting in significant bias when generalizing to the broader user base. Additionally, the complexity of industrial processes reduces the quality of RCT data, which is often subject to heterogeneity from potential interference and selection bias, making it difficult to correct. Moreover, existing data fusion methods are challenging to implement effectively in complex industrial settings due to the high dimensionality of features and the strict assumptions that are hard to verify with real-world data. To address these issues, we propose an empirical data fusion method called pseudo-sample matching. By generating pseudo-samples from biased, low-quality RCT data and matching them with the most similar samples from large-scale observational data, the method expands the RCT dataset while mitigating its heterogeneity. We validated the method through simulation experiments, conducted offline and online tests using real-world data. In a week-long online experiment, we achieved a 0.41% improvement in profit, which is a considerable gain when scaled to industrial scenarios with hundreds of millions in revenue. In addition, we discuss the harm to model training, offline evaluation, and online economic benefits when the RCT data quality is not high, and emphasize the importance of improving RCT data quality in industrial scenarios.

Introduction

In the competitive ride-hailing market, companies rely on uplift models to optimize discount strategies. These models aim to identify "Persuadables"—users whose behavior is positively changed by a discount—while avoiding "Sure Things" (who book anyway) and "Sleeping Dogs" (who might be negatively impacted). Traditionally, these models are trained on data from Randomized Controlled Trials (RCTs). However, RCTs in industrial settings are often small (e.g., 0.65% of traffic) and suffer from inherent biases due to complex engineering constraints and link interference between drivers and passengers.

Methodology: Pseudo-Sample Matching

To overcome the limitations of small, biased RCTs, we propose Pseudo-Sample Matching. This empirical data fusion method uses biased RCT data as a seed to extract high-quality samples from large-scale observational data (OBS).

The Pseudo-Sample Matching workflow: from biased RCT to a balanced 'Good RCT' using observational data.

The process involves three main steps:

  1. Bias Identification: Calculating the feature mean difference between the RCT treatment groups and the entire population to identify bias vectors.
  2. Pseudo-Sample Generation: Creating "ideal" versions of RCT samples by adjusting them with a bias vector and a fusion ratio ($k$) to align them with the overall distribution.
  3. Matching and Fusion: Using K-Nearest Neighbors (KNN) to find the $k$ most similar real-world observations in the massive OBS dataset. To ensure scalability, we employ a "bucketing" strategy, grouping data by critical features like supply-demand ratios before matching on a distributed cluster.

Experimental Results

The method was validated through both synthetic simulations and real-world industrial deployment at DiDi Chuxing.

Comparison of feature distributions across different data sources (biased RCT, unbiased RCT, OBS) and the final fused dataset.

Simulation and Offline Evaluation

In simulations with 20-dimensional features, our method consistently outperformed baseline models like TARNet and Causal Forests in terms of Qini coefficients and Mean Absolute Percentage Error (MAPE). A critical insight was that relying solely on biased RCTs for evaluation can be misleading; as the fusion ratio increases, traditional offline metrics might appear to degrade even as the model's true performance on ground-truth data improves.

Online Impact

In a week-long online experiment across 62 major cities in China, the model trained with fused data achieved:

  • +0.87% in Gross Merchandise Value (GMV).
  • +0.41% in overall profit.

While 0.41% sounds small, it represents a considerable gain when scaled to industrial scenarios involving hundreds of millions in revenue.

Conclusion

Pseudo-Sample Matching provides a robust, scalable way to leverage massive observational datasets to correct and augment small, biased RCTs. While it requires careful feature selection and computational resources for KNN matching, its ability to improve real-world economic outcomes without strict theoretical assumptions makes it highly valuable for industrial applications. Future work will focus on automated ways to improve RCT data quality directly at the source.

Further details and simulation code are available at: https://github.com/Kairong-Han/Pseudo-Matching

Summary updated

Abstract

In the online ride-hailing pricing context, companies often conduct randomized controlled trials (RCTs) and utilize uplift models to assess the effect of discounts on customer orders, which substantially influences competitive market outcomes. However, due to the high cost of RCTs, the proportion of trial data relative to observational data is small, which only accounts for 0.65\% of total traffic in our context, resulting in significant bias when generalizing to the broader user base. Additionally, the complexity of industrial processes reduces the quality of RCT data, which is often subject to heterogeneity from potential interference and selection bias, making it difficult to correct. Moreover, existing data fusion methods are challenging to implement effectively in complex industrial settings due to the high dimensionality of features and the strict assumptions that are hard to verify with real-world data. To address these issues, we propose an empirical data fusion method called pseudo-sample matching. By generating pseudo-samples from biased, low-quality RCT data and matching them with the most similar samples from large-scale observational data, the method expands the RCT dataset while mitigating its heterogeneity. We validated the method through simulation experiments, conducted offline and online tests using real-world data. In a week-long online experiment, we achieved a 0.41\% improvement in profit, which is a considerable gain when scaled to industrial scenarios with hundreds of millions in revenue. In addition, we discuss the harm to model training, offline evaluation, and online economic benefits when the RCT data quality is not high, and emphasize the importance of improving RCT data quality in industrial scenarios. Further details of the simulation experiments can be found in the GitHub repository https://github.com/Kairong-Han/Pseudo-Matching.
BibTeX
@inproceedings{han2025augmenting,
  author = {Kairong Han and Weidong Huang and Taiyang Zhou and Peng Zhen and Kun Kuang},
  title = {{Augmenting Limited and Biased RCTs through Pseudo-Sample Matching-Based Observational Data Fusion Method}},
  booktitle = {ACM International Conference on Information and Knowledge Management},
  year = {2025},
  doi = {10.1145/3746252.3761532},
}
Figure