This summary provides a technical overview of the perspective paper "From More to Better: Causal Inference and Digital Experimentation in the Age of AI," which synthesizes insights from the 2026 Causal Inference and Digital Experimentation Roundtable (CIDER).
TL;DR
As AI lowers the barrier to generating data and running analyses, the bottleneck for organizations shifts from "doing the work" to "judging the work." This paper argues that data scientists must move beyond individual A/B tests to focus on building cumulative organizational knowledge, validating synthetic evidence (like LLM-based "digital twins"), and designing evaluation frameworks that can keep pace with rapidly evolving, non-deterministic AI products.
What Problem Does This Paper Address?
We are entering an era of "analytical abundance." AI agents can now write SQL, execute experiments, and summarize results at a scale previously impossible. However, more analysis does not automatically lead to better decisions.
The authors identify a critical shift in the leverage of data scientists. In the past, the challenge was the technical execution of causal inference. Today, the challenge is assessing the credibility, relevance, and generalizability of findings. The paper tackles three specific tensions:
- Learning vs. Launching: Organizations often treat experiments as simple "go/no-go" gates, losing the underlying "why" that could inform future strategy.
- Synthetic vs. Real Evidence: As LLMs allow us to simulate user behavior, when can we trust these "synthetic respondents" over expensive live-user tests?
- Static vs. Evolving Products: Traditional metrics fail when AI products are open-ended and user behavior adapts dynamically to the model's capabilities.
Methods
This is a position paper derived from the CIDER 2026 roundtable, which brought together practitioners from tech giants (OpenAI, Netflix, Meta, etc.) and academics (UC Berkeley, Cornell).
The paper distills their collective experience into three strategic pillars:
- Mechanistic Reasoning: Prioritizing "mechanism metrics" (mediators) to understand the causal path from treatment to outcome.
- The Evidence Ladder: A framework for combining diverse evidence sources—ranging from low-cost synthetic screening and offline evaluations to high-stakes live experiments and long-term holdbacks.
- Hybrid Inference: Utilizing techniques like Prediction-Powered Inference (PPI) to combine many model predictions with small samples of ground-truth data to correct for bias while maintaining statistical power.
Datasets
While the paper does not introduce a new training dataset, it discusses the evaluation of several key benchmarks and datasets used in the field:
- Twin-2K-500: A dataset introduced by Toubia et al. (2025) for building "digital twins" of over 2,000 people. The authors note that while these twins predict individual responses well, they often fail to replicate complex experimental patterns.
- Internal Industry Frameworks: The paper references specific methodologies used at Netflix (structural models for cumulative impact) and Waymo (high-stakes simulation for rare-event detection in self-driving).
Key Insights and Ideas
- Metadata is the New Gold: To learn across experiments, organizations must record not just the p-value, but the intent, the causal diagram hypothesized by the experimenter, and the context (e.g., model version). AI can help by extracting this structure from unstructured documents and code changes.
- The Fidelity Gap: LLMs are surprisingly good at mimicking human survey responses (e.g., political views), but they struggle to reproduce human choices in strategic, multi-step decision tasks.
- Evaluation Drift: An experiment on a coding agent might show success for "routine debugging," but once deployed, users might shift to using it for "complex migrations." If the evaluation set doesn't evolve with user behavior, the metrics become "zombies"—technically correct but practically irrelevant.
- The Junior DS Crisis: A "shared concern" among the authors is that AI might automate the "grunt work" (basic querying and analysis) through which junior data scientists traditionally build intuition and domain expertise.
Key Results
The paper categorizes its findings by the level of agreement among the experts:
- Consensus: The value of experimental metadata is undisputed for organizational learning.
- Shared: AI products present unique evaluation challenges that require moving beyond aggregate measures to qualitative evidence like screenshots and human review.
- Contested: There is significant debate over whether synthetic evidence can ever truly replace live experiments, and how much "structure" (like mandatory causal diagrams) should be forced upon experimenters versus allowing for flexible, decentralized analysis.
Limitations and Future Work
The authors highlight several "evergreen" challenges that remain unresolved:
- Incentive Compatibility: How do we encourage teams to document "failed" experiments and rich metadata when the organizational pressure is solely on "launching"?
- Leakage in Synthetic Evidence: Ensuring that the experimental results we are trying to predict haven't already "leaked" into the training data of the LLMs being used as synthetic respondents.
- Metric Proliferation: As we add more guardrails and task-specific evaluations for AI, how do we prevent "metric fatigue" and keep a clear line of sight to "North Star" goals like long-term retention.
Summary updated