Abstract
Large Language Models (LLMs) enable a new form of digital experimentation where treatments combine human and model-generated content in increasingly sophisticated ways. The main methodological challenge in this setting is representing these high-dimensional treatments without losing their semantic meaning or rendering analysis intractable. Here, we address this problem by focusing on learning low-dimensional representations that capture the underlying structure of such treatments. These representations enable downstream applications such as guiding generative models to produce meaningful treatment variants and facilitating adaptive assignment in online experiments. We propose double kernel representation learning, which models the causal effect through the inner product of kernel-based representations of treatments and user covariates. We develop an alternating-minimization algorithm that learns these representations efficiently from data and provides convergence guarantees under a low-rank factor model. As an application of this framework, we introduce an adaptive design strategy for online experimentation and demonstrate the method's effectiveness through numerical experiments.
Introduction
In the era of Generative AI, the bottleneck of digital experimentation has shifted from content creation to evaluation. We can now effortlessly generate thousands of "treatments"—such as different ad headlines, email subject lines, or product images. However, traditional randomized controlled trials (A/B tests) treat every variant as a discrete, unrelated "bucket." This creates two major hurdles: statistical power is diluted across too many buckets, and the semantic structure between treatments (e.g., two headlines with similar meanings) is ignored.
This paper introduces Double Kernel Representation Learning (DKRL), a framework that leverages semantic similarity encoded in LLM embeddings to pool information across similar treatments and users, enabling more efficient causal inference and smarter adaptive experimentation.
Methodology: Double Kernel Representation Learning
The core innovation of DKRL is modeling the treatment effect as the inner product of two low-dimensional representations: one for the treatment (e.g., a headline embedding) and one for the user (covariates). Instead of simple linear models, DKRL utilizes Reproducing Kernel Hilbert Spaces (RKHS), allowing it to capture complex, non-linear relationships in high-dimensional embedding spaces.
To learn these representations, the authors developed an alternating-minimization algorithm. This approach iteratively updates treatment features while holding user features constant, and vice versa. This method is not only theoretically sound with convergence guarantees but also computationally efficient.
As shown in the complexity analysis above, the algorithm's efficiency stems from its reliance on kernel inversions and iterative ridge updates, which scale predictably compared to the heavy computational overhead of deep learning alternatives.
Experimental Setup
The method was evaluated using "semi-synthetic" experiments that utilize real-world features and embeddings while simulating outcomes to provide a ground truth for causal effects. The datasets included:
- Upworthy Research Archive: Real-world A/B tests for news headlines.
- MIND Dataset: Microsoft News dataset for click-through rate prediction.
- ASOS E-Commerce Dataset: An open-source A/B testing dataset from the fashion retailer ASOS.
Key Results and Insights
DKRL consistently outperformed neural-net-based baselines (such as Structured Intervention Networks) and standard product-kernel regressions across the Upworthy and ASOS datasets, achieving lower training and testing errors.
Efficiency and Adaptivity
- Borrowing Strength: By using kernels, the model "borrows" information from similar treatments, effectively increasing the sample size for any given "type" of treatment.
- Computational Speed: On a standard laptop (M2 Max chip), DKRL solved complex estimation problems in fractions of a second, whereas deep learning baselines often required minutes.
- Sublinear Regret: In online bandit settings, the DKRL-based adaptive strategy demonstrated "sublinear regret," meaning the cost of suboptimal assignments decreases as the model learns.
Interpretability
The learned low-dimensional features often correspond to human-interpretable concepts. For instance, in the Upworthy headlines, the model identified dimensions like "narrative surprise" and "conflict-driven tone" as primary drivers of user engagement.
Conclusion and Future Work
DKRL provides a robust framework for modern experimentation with high-dimensional, AI-generated content. While the current model assumes a fixed-basis setting and no interference between users (SUTVA), future work could extend these guarantees to infinite-dimensional bases and social network contexts. Additionally, as the method relies heavily on the quality of initial LLM embeddings, further research into embedding robustness and the ethical implications of automated experimentation is warranted.
Summary updated