TL;DR
Uplift modeling—predicting how much an individual's behavior changes due to an intervention—is increasingly using Large Language Models (LLMs) to understand complex text features. However, if the text contains "leaked" information about the treatment (like adding "Clearance" to a product title after a discount is decided), models learn brittle shortcuts instead of true causal effects. This paper introduces LULR, a regularization method that uses information theory to "scrub" treatment leakage from model representations, leading to more robust predictions and significant gains in real-world industrial pricing.
What Problem Does This Paper Address?
In fields like marketing and pricing, "uplift modeling" helps decide which customers or products should receive a treatment (like a coupon or a discount) to maximize the return on investment. While LLMs are great at extracting semantic nuances from product titles (e.g., distinguishing an "essential daily item" from a "niche luxury good"), they are prone to shortcut learning.
If a dataset includes text updated after a treatment was applied, that text might "leak" the treatment status. For example, if a model sees the word "Sale" in a product description, it might correctly predict a high purchase rate, but it's doing so because it sees the result of the discount, not because it understands how the product's demand responds to a price change. This shortcut fails in deployment when the model needs to make a decision before the "Sale" tag exists.
Methods
The authors propose LLM-based Uplift Leakage Regularization (LULR). The architecture consists of a frozen LLM encoder paired with a trainable "causal head" (based on the DragonNet architecture).
The core innovation is the addition of an information-bottleneck-inspired penalty. Specifically, they minimize the Mutual Information (MI) between the model's internal shared representation ($s$) and the treatment ($T$). To make this mathematically tractable and differentiable, they use the Contrastive Log-ratio Upper Bound (CLUB) estimator.
By penalizing $I(s; T)$, the model is forced to find features that describe the unit's characteristics without simply "sniffing out" whether it was treated.
Datasets
The researchers evaluated LULR across several settings:
- Industrial Dataset: A massive product-pricing Randomized Controlled Trial (RCT) from JD.com (~1 million units) featuring real product titles and GMV (Gross Merchandise Value) outcomes.
- Public Benchmarks: The Hillstrom (email marketing) and x5 datasets, used primarily to test performance on structured data.
- Semi-synthetic (semi_text): A constructed dataset where text is intentionally designed to carry a signal to test the model's ability to handle semantic features.
Key Insights and Ideas
- Targeted vs. Generic Regularization: Standard techniques like weight decay (L2 regularization) shrink all parameters equally. LULR is "surgical"—it specifically targets the dependence on treatment information. The authors found that weight decay failed to improve scores in the presence of leakage, whereas LULR succeeded.
- Representational Scrubbing: The penalty is applied to the shared representation before it splits into separate "heads" for treated and control outcomes. This ensures the foundational features the model uses are "clean" of treatment leakage.
- Shortcut Learning as a Causal Risk: The paper frames shortcut learning not just as an overfitting problem, but as a fundamental risk to causal identification when using high-dimensional inputs like text.
Key Results
- Controlled Proxy Experiment: When leakage was intentionally injected into the data, LULR significantly improved the model's ranking ability (measured by the Qini coefficient) compared to a baseline with no regularization.
- Industrial Offline Performance: In offline tests on JD.com data, LULR improved the mean [email protected] (a measure of how well the model identifies top responders) from 0.106 to 0.258.
- Online A/B Test: In a one-week live deployment for product pricing, the LULR-based policy achieved a 5.45% increase in sales volume and a 1.91% increase in GMV relative to the control group.
Limitations and Future Work
- Backbone Choice: On some public structured-only datasets, the neural models (even with LULR) still lagged behind traditional Gradient Boosted Decision Trees (GBDT), suggesting that the choice of model architecture remains a critical factor.
- Over-balancing Risk: There is a trade-off; if the regularization strength ($\beta$) is too high, the model might discard useful predictive features that happen to correlate slightly with the treatment in a small sample.
- Long-term Dynamics: The online results are short-term (one week). Future work is needed to see if these gains hold up over months as market conditions and product vocabularies shift.
Summary updated