Improve ROI with Causal Learning and Conformal Prediction

Meng Ai, Zhuo Chen, Jibin Wang, Jing Shang, Tao Tao, Zhen Li

TL;DR

This paper introduces a robust Direct ROI Prediction (rDRP) method to improve the reliability of decision-making in commercial applications like advertising and marketing. By combining conformal prediction and Monte Carlo dropout, rDRP effectively addresses the challenges of covariate shift and insufficient data that plague existing state-of-the-art methods, leading to significant improvements in return on investment (ROI).

What Problem Does This Paper Address?

In many commercial sectors, optimizing resource allocation to maximize Return on Investment (ROI) is crucial. This often boils down to the Cost-aware Binary Treatment Assignment Problem (C-BTAP), where the goal is to decide which individuals should receive a "treatment" (e.g., an advertisement, a discount) to maximize overall profit, given a budget constraint. The current state-of-the-art method for this, Direct ROI Prediction (DRP), uses neural networks to directly predict an individual's ROI. However, DRP struggles in real-world scenarios due to two main issues:

  1. Covariate Shift: The distribution of data in the real-world deployment (test set) can differ significantly from the data used to train the model. For instance, a model trained on weekday user data might perform poorly on weekend data if user behavior changes drastically.
  2. Insufficient Training Data: Real-world experiments (like A/B tests or RCTs) to collect training data are often limited in scale and duration due to cost or potential impact on user experience. This scarcity of data makes it difficult for models like DRP to converge properly and generalize well.

These limitations can lead to unreliable ROI predictions and suboptimal resource allocation, hindering the effectiveness of intelligent decision-making systems.

Methods

The paper proposes a robust Direct ROI Prediction (rDRP) method that builds upon the existing DRP model without altering its core structure or requiring retraining. rDRP introduces a post-processing stage that leverages uncertainty quantification to calibrate the DRP's point estimates. The key components are:

  • Monte Carlo (MC) Dropout: This technique is used during inference to estimate the standard deviation (uncertainty) of the DRP's ROI prediction. By applying dropout multiple times to the same input, it generates a distribution of predictions, from which the standard deviation can be calculated. This is a more efficient way to estimate uncertainty compared to ensemble methods.
  • Conformal Prediction (CP): CP is employed to generate rigorous prediction intervals for the ROI. Unlike traditional methods that require separate predictions for revenue and cost uplift, rDRP directly estimates intervals for the ROI. This is achieved by defining a "Conformal Score" that combines the DRP's point estimate, its estimated standard deviation, and a derived "true" ROI value (approximated by the convergence point of the DRP's loss function). CP guarantees that the prediction interval will contain the true ROI with a user-specified probability (e.g., 90%).
  • Heuristic Calibration: Inspired by a Kaggle competition, the paper proposes a heuristic method to calibrate the DRP's point estimates using the prediction intervals. This involves combining the point estimate with the interval information in several ways (e.g., weighting by the upper bound or width of the interval) and selecting the best performing method on a calibration set.

The overall rDRP process involves training the DRP model, then using a small, separate RCT dataset (calibration set) to estimate the convergence point ROI, calculate uncertainty using MC dropout, generate prediction intervals using CP, and finally calibrate the point estimates.

Datasets

The paper evaluates its method on three public real-world datasets commonly used for uplift modeling:

  • CRITEO-UPLIFT v2: A large-scale dataset from an advertising scene.
  • Meituan-LIFT: A dataset from a smart coupon marketing campaign for food delivery.
  • Alibaba-LIFT: Another large-scale RCT dataset from Alibaba's advertising platform.

These datasets are used in four experimental settings: sufficient data with no covariate shift (SuNo), sufficient data with covariate shift (SuCo), insufficient data with no covariate shift (InNo), and insufficient data with covariate shift (InCo).

Key Insights and Ideas

  • Bridging the Gap Between Theory and Practice: The paper directly tackles the practical limitations (covariate shift, insufficient data) that prevent theoretical models like DRP from performing optimally in real-world commercial applications.
  • Uncertainty as a Feature: Instead of just predicting a single ROI value, rDRP leverages the uncertainty in those predictions (via MC dropout and CP) to improve their accuracy and robustness. This is a crucial shift from point estimates to more reliable interval estimates.
  • Conformal Prediction for ROI: Applying conformal prediction directly to ROI, rather than its constituent parts (revenue/cost uplift), is a novel approach that simplifies the process and avoids error propagation from combining separate uplift models.
  • Heuristic Calibration's Effectiveness: The inspiration from a Kaggle competition highlights how practical, albeit heuristic, methods can effectively bridge the gap between theoretical interval estimates and actionable point estimates for decision-making.

Key Results

  • Significant Improvement: rDRP consistently outperforms the state-of-the-art DRP model and other baseline methods (TPM, DR) across various offline test settings, particularly in scenarios with insufficient data and covariate shift.
  • Offline AUCC Gains: The paper reports detailed Area Under Cost Curve (AUCC) results, showing that rDRP achieves higher scores, indicating better ROI ranking performance. For instance, in the "Insufficient data and Covariate shift" setting on the CRITEO-UPLIFT v2 dataset, rDRP shows a notable improvement over DRP.
  • Online A/B Test Validation: Online A/B tests conducted on a short video platform demonstrate that rDRP leads to significant improvements in advertising revenue compared to DRP and a random control group, especially in challenging scenarios (SuCo, InNo, InCo).
  • Ablation Study Validation: Ablation studies confirm the individual contributions of MC dropout and Conformal Prediction, showing that each component enhances the performance of the base DRP model.

Limitations and Future Work

The authors acknowledge several limitations:

  • Binary Treatment Only: rDRP is currently designed for binary treatments (presence/absence of intervention). Extending it to multiple treatments would require methods like "Divide and Conquer."
  • RCT Data Requirement: Like DRP, rDRP requires data from Randomized Controlled Trials (RCTs) and cannot directly use observational datasets.
  • Inference Latency: The multiple inferences required for rDRP during the online deployment phase could lead to latency if computational resources are limited. Balancing resource costs with business benefits is crucial.
  • Positive Treatment Effect Assumption: rDRP assumes that marketing interventions have a positive influence on response and entail a positive cost.
  • Conformalizing Scalar Uncertainty Estimates: The paper notes a potential theoretical disconnect where uncertainty scalars might not scale perfectly with the error rate, potentially affecting the precise control over prediction interval width.

Future work includes exploring rigorous interval estimation for neural network-based uplift models on non-RCT data, extending to multiple treatment scenarios without relying solely on divide-and-conquer, and developing more sophisticated calibration methods beyond heuristics.

Summary updated

Abstract

In the commercial sphere, such as operations and maintenance, advertising, and marketing recommendations, intelligent decision-making utilizing data mining and neural network technologies is crucial, especially in resource allocation to optimize ROI. This study delves into the Cost-aware Binary Treatment Assignment Problem (C-BTAP) across different industries, with a focus on the state-of-the-art Direct ROI Prediction (DRP) method. However, the DRP model confronts issues like covariate shift and insufficient training data, hindering its real-world effectiveness. Addressing these challenges is essential for ensuring dependable and robust predictions in varied operational contexts. This paper presents a robust Direct ROI Prediction (rDRP) method, designed to address challenges in real-world deployment of neural network-based uplift models, particularly under conditions of covariate shift and insufficient training data. The rDRP method, enhancing the standard DRP model, does not alter the model's structure or require retraining. It utilizes conformal prediction and Monte Carlo dropout for interval estimation, adapting to model uncertainty and data distribution shifts. A heuristic calibration method, inspired by a Kaggle competition, combines point and interval estimates. The effectiveness of these approaches is validated through offline tests and online A/B tests in various settings, demonstrating significant improvements in target rewards compared to the state-of-the-art method.
BibTeX
@misc{ai2024improve,
  author = {Meng Ai and Zhuo Chen and Jibin Wang and Jing Shang and Tao Tao and Zhen Li},
  title = {{Improve ROI with Causal Learning and Conformal Prediction}},
  year = {2024},
  eprint = {2407.01065},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.01065},
}
Figure