Regularizing Treatment Leakage for LLM-based Uplift Estimation

Bojing Feng, Jinqiu Deng, Zihan Hong, Yuan Xue, Wenqi Xiang, Shuhan Wang, Feng Xiao, Shiwei Wang, Zijiang Liu, Shaomin Li, Yongzhi Qi

TL;DR

Uplift modeling—predicting how much an individual's behavior changes due to an intervention—is increasingly using Large Language Models (LLMs) to understand complex text features. However, if the text contains "leaked" information about the treatment (like adding "Clearance" to a product title after a discount is decided), models learn brittle shortcuts instead of true causal effects. This paper introduces LULR, a regularization method that uses information theory to "scrub" treatment leakage from model representations, leading to more robust predictions and significant gains in real-world industrial pricing.

What Problem Does This Paper Address?

In fields like marketing and pricing, "uplift modeling" helps decide which customers or products should receive a treatment (like a coupon or a discount) to maximize the return on investment. While LLMs are great at extracting semantic nuances from product titles (e.g., distinguishing an "essential daily item" from a "niche luxury good"), they are prone to shortcut learning.

If a dataset includes text updated after a treatment was applied, that text might "leak" the treatment status. For example, if a model sees the word "Sale" in a product description, it might correctly predict a high purchase rate, but it's doing so because it sees the result of the discount, not because it understands how the product's demand responds to a price change. This shortcut fails in deployment when the model needs to make a decision before the "Sale" tag exists.

Methods

The authors propose LLM-based Uplift Leakage Regularization (LULR). The architecture consists of a frozen LLM encoder paired with a trainable "causal head" (based on the DragonNet architecture).

The core innovation is the addition of an information-bottleneck-inspired penalty. Specifically, they minimize the Mutual Information (MI) between the model's internal shared representation ($s$) and the treatment ($T$). To make this mathematically tractable and differentiable, they use the Contrastive Log-ratio Upper Bound (CLUB) estimator.

The LULR architecture showing how the CLUB penalty regularizes the shared representation during training.

By penalizing $I(s; T)$, the model is forced to find features that describe the unit's characteristics without simply "sniffing out" whether it was treated.

Datasets

The researchers evaluated LULR across several settings:

  • Industrial Dataset: A massive product-pricing Randomized Controlled Trial (RCT) from JD.com (~1 million units) featuring real product titles and GMV (Gross Merchandise Value) outcomes.
  • Public Benchmarks: The Hillstrom (email marketing) and x5 datasets, used primarily to test performance on structured data.
  • Semi-synthetic (semi_text): A constructed dataset where text is intentionally designed to carry a signal to test the model's ability to handle semantic features.
Summary of datasets used in the study.

Key Insights and Ideas

  • Targeted vs. Generic Regularization: Standard techniques like weight decay (L2 regularization) shrink all parameters equally. LULR is "surgical"—it specifically targets the dependence on treatment information. The authors found that weight decay failed to improve scores in the presence of leakage, whereas LULR succeeded.
  • Representational Scrubbing: The penalty is applied to the shared representation before it splits into separate "heads" for treated and control outcomes. This ensures the foundational features the model uses are "clean" of treatment leakage.
  • Shortcut Learning as a Causal Risk: The paper frames shortcut learning not just as an overfitting problem, but as a fundamental risk to causal identification when using high-dimensional inputs like text.

Key Results

  • Controlled Proxy Experiment: When leakage was intentionally injected into the data, LULR significantly improved the model's ranking ability (measured by the Qini coefficient) compared to a baseline with no regularization.
  • Industrial Offline Performance: In offline tests on JD.com data, LULR improved the mean [email protected] (a measure of how well the model identifies top responders) from 0.106 to 0.258.
  • Online A/B Test: In a one-week live deployment for product pricing, the LULR-based policy achieved a 5.45% increase in sales volume and a 1.91% increase in GMV relative to the control group.
Comparison of LULR against no regularization and weight decay under injected leakage conditions.

Limitations and Future Work

  • Backbone Choice: On some public structured-only datasets, the neural models (even with LULR) still lagged behind traditional Gradient Boosted Decision Trees (GBDT), suggesting that the choice of model architecture remains a critical factor.
  • Over-balancing Risk: There is a trade-off; if the regularization strength ($\beta$) is too high, the model might discard useful predictive features that happen to correlate slightly with the treatment in a small sample.
  • Long-term Dynamics: The online results are short-term (one week). Future work is needed to see if these gains hold up over months as market conditions and product vocabularies shift.

Summary updated

Abstract

Large language models (LLMs) provide semantic features for uplift estimation, but treatment-revealing text can induce prediction shortcuts that do not transfer to deployment. We propose \emph{LLM-based Uplift Leakage Regularization} (\textsc{LULR}) to address this risk. With the LLM encoder frozen, \textsc{LULR} penalizes dependence between the causal head's trainable shared representation and treatment while retaining separate outcome heads. The penalty uses a variational approximation to the contrastive log-ratio upper bound (CLUB) of mutual information, targeting treatment dependence rather than generic parameter shrinkage. Controlled proxy experiments provide exploratory evidence of improved ranking scores under injected treatment information relative to no regularization and the tested weight-decay setting. Structured-only benchmarks show modest, dataset-dependent effects. On an industrial pricing randomized controlled trial, mean offline $\text{lift@}0.3$ is $0.258$ for \textsc{LULR}, compared with $0.106$ for the same-backbone baseline. A separate one-week online A/B test of the complete deployed pricing policy records $5.45\%$ higher sales volume and $1.91\%$ higher gross merchandise value than the control arm. These short-term policy-level gains do not isolate the regularizer's effect.
BibTeX
@misc{feng2026regularizing,
  author = {Bojing Feng and Jinqiu Deng and Zihan Hong and Yuan Xue and Wenqi Xiang and Shuhan Wang and Feng Xiao and Shiwei Wang and Zijiang Liu and Shaomin Li and Yongzhi Qi},
  title = {{Regularizing Treatment Leakage for LLM-based Uplift Estimation}},
  year = {2026},
  eprint = {2611.20794},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20794},
}
Figure