Overview
This paper argues that the "expertise" of decision-makers—such as doctors prescribing medicine or teachers assigning interventions—should be treated as a valuable information source rather than merely a source of confounding in Treatment Effect Estimation (TEE). By formally defining predictive and prognostic expertise using information theory, the authors demonstrate that identifying the type of expertise in a dataset allows researchers to select the most effective machine learning models for estimating personalized treatment effects.
The Challenge: Confounding vs. Expertise
In the realm of causal inference, we frequently attempt to answer "what-if" questions: If I give this patient Drug A instead of Drug B, what will happen? The fundamental problem of causal inference is that we never observe both outcomes for the same individual simultaneously.
Most machine learning approaches for TEE treat the decision-maker’s policy (how they choose who gets which treatment) as a nuisance. Because experts assign treatments based on patient features, they create "covariate shift" or "confounding," making the treated group look fundamentally different from the untreated group. Standard methods attempt to "balance" the data to mimic a randomized controlled trial.
The authors argue this perspective represents a missed opportunity. If an expert prescribes a treatment, that action inherently encodes their domain knowledge. Instead of merely trying to "undo" the expert's bias, we can leverage the fact that they are an expert as an inductive bias to improve our predictive models.
Formalizing Expertise
To operationalize this concept, the authors introduce a formal mathematical framework based on Shannon Entropy. They define expertise as the proportion of "randomness" in an action that can be explained by knowing the potential outcomes.
They distinguish between two specific types of expertise:
- Predictive Expertise: The decision-maker chooses actions based specifically on the treatment effect (the difference between potential outcomes). For example, a doctor choosing the drug that provides the maximum specific "gain" for a patient.
- Prognostic Expertise: The decision-maker chooses actions based on the overall likely outcome, regardless of the specific treatment effect. For example, a teacher giving extra attention to students who are generally struggling, even if the marginal benefit of the explanation is identical for everyone.
The "Expertise-Informed" Pipeline: Because the exact expertise type of a historical dataset is usually unknown, the authors propose a two-step pipeline:
- Step 1: Use an "Action-predictive" model (like DragonNet) to estimate the levels of predictive and prognostic expertise present in the data.
- Step 2: Based on the ratio of these estimates, select the best downstream TEE method. If expertise is primarily predictive, use action-predictive models; if it is prognostic, use balancing models (like CFRNet).
Theoretical Insights
A core contribution of the paper is proving that expertise is a double-edged sword.
As shown in Proposition 1, higher expertise necessarily leads to poorer "overlap" (the treated and untreated groups look less alike). This makes TEE mathematically harder because there are fewer comparable examples. However, as the figure illustrates, in these high-expertise/low-overlap scenarios, leveraging the expert's "signal" becomes absolutely critical for accurate estimation.
Furthermore, the authors show that popular "balancing" methods can actually be harmful. These methods work by trying to make treatment assignment look random. If an expert possesses high predictive expertise, balancing actively discards the most useful information in the dataset, leading to degraded performance.
Empirical Findings
To test these theories, the authors utilized semi-synthetic benchmarks (like TCGA and News datasets) where they could control the "expert" policy and observe ground-truth treatment effects.
The results clearly demonstrate that model selection matters. As seen in the performance graphs, in environments with high predictive expertise, Action-predictive methods (red line) significantly improved over the baseline. Conversely, in prognostic settings, Balancing methods (blue line) proved superior.
Ultimately, their automated "Expertise-informed" pipeline achieved "best-of-both-worlds" performance. By automatically choosing the model based on estimated expertise, it outperformed standard baselines, propensity-weighting, and balancing across all dataset types.
Limitations & Future Directions
- The Non-Linear Gap: In experiments with complex non-linear interactions, the "Expertise-informed" pipeline struggled slightly more, primarily because the underlying "Balancing" models performed poorly in non-linear settings. The authors suggest this is likely an implementation artifact rather than a theoretical flaw.
- Oracle Dependency: Estimating expertise relies heavily on the quality of the initial outcome model. If this model is poor, the expertise estimate will be biased, potentially leading to incorrect model selection.
- Real-World Validation: While the theoretical and semi-synthetic results are robust, validating this framework on purely real-world data—where counterfactuals are truly unobservable—remains a significant challenge.
Ultimately, this framework encourages practitioners to deeply consider why a treatment was assigned before blindly applying a causal inference algorithm.
Summary updated