Defining Expertise: Applications to Treatment Effect Estimation

Alihan Hüyük, Qiyao Wei, Alicia Curth, Mihaela van der Schaar

Overview

This paper argues that the "expertise" of decision-makers—such as doctors prescribing medicine or teachers assigning interventions—should be treated as a valuable information source rather than merely a source of confounding in Treatment Effect Estimation (TEE). By formally defining predictive and prognostic expertise using information theory, the authors demonstrate that identifying the type of expertise in a dataset allows researchers to select the most effective machine learning models for estimating personalized treatment effects.

The Challenge: Confounding vs. Expertise

In the realm of causal inference, we frequently attempt to answer "what-if" questions: If I give this patient Drug A instead of Drug B, what will happen? The fundamental problem of causal inference is that we never observe both outcomes for the same individual simultaneously.

Most machine learning approaches for TEE treat the decision-maker’s policy (how they choose who gets which treatment) as a nuisance. Because experts assign treatments based on patient features, they create "covariate shift" or "confounding," making the treated group look fundamentally different from the untreated group. Standard methods attempt to "balance" the data to mimic a randomized controlled trial.

The authors argue this perspective represents a missed opportunity. If an expert prescribes a treatment, that action inherently encodes their domain knowledge. Instead of merely trying to "undo" the expert's bias, we can leverage the fact that they are an expert as an inductive bias to improve our predictive models.

Formalizing Expertise

To operationalize this concept, the authors introduce a formal mathematical framework based on Shannon Entropy. They define expertise as the proportion of "randomness" in an action that can be explained by knowing the potential outcomes.

They distinguish between two specific types of expertise:

  1. Predictive Expertise: The decision-maker chooses actions based specifically on the treatment effect (the difference between potential outcomes). For example, a doctor choosing the drug that provides the maximum specific "gain" for a patient.
  2. Prognostic Expertise: The decision-maker chooses actions based on the overall likely outcome, regardless of the specific treatment effect. For example, a teacher giving extra attention to students who are generally struggling, even if the marginal benefit of the explanation is identical for everyone.

The "Expertise-Informed" Pipeline: Because the exact expertise type of a historical dataset is usually unknown, the authors propose a two-step pipeline:

  • Step 1: Use an "Action-predictive" model (like DragonNet) to estimate the levels of predictive and prognostic expertise present in the data.
  • Step 2: Based on the ratio of these estimates, select the best downstream TEE method. If expertise is primarily predictive, use action-predictive models; if it is prognostic, use balancing models (like CFRNet).

Theoretical Insights

A core contribution of the paper is proving that expertise is a double-edged sword.

The theoretical trade-off between action variability (overlap) and expertise.

As shown in Proposition 1, higher expertise necessarily leads to poorer "overlap" (the treated and untreated groups look less alike). This makes TEE mathematically harder because there are fewer comparable examples. However, as the figure illustrates, in these high-expertise/low-overlap scenarios, leveraging the expert's "signal" becomes absolutely critical for accurate estimation.

Furthermore, the authors show that popular "balancing" methods can actually be harmful. These methods work by trying to make treatment assignment look random. If an expert possesses high predictive expertise, balancing actively discards the most useful information in the dataset, leading to degraded performance.

Empirical Findings

To test these theories, the authors utilized semi-synthetic benchmarks (like TCGA and News datasets) where they could control the "expert" policy and observe ground-truth treatment effects.

Performance of different TEE methods across varying levels of predictive and prognostic expertise.

The results clearly demonstrate that model selection matters. As seen in the performance graphs, in environments with high predictive expertise, Action-predictive methods (red line) significantly improved over the baseline. Conversely, in prognostic settings, Balancing methods (blue line) proved superior.

Ultimately, their automated "Expertise-informed" pipeline achieved "best-of-both-worlds" performance. By automatically choosing the model based on estimated expertise, it outperformed standard baselines, propensity-weighting, and balancing across all dataset types.

Limitations & Future Directions

  • The Non-Linear Gap: In experiments with complex non-linear interactions, the "Expertise-informed" pipeline struggled slightly more, primarily because the underlying "Balancing" models performed poorly in non-linear settings. The authors suggest this is likely an implementation artifact rather than a theoretical flaw.
  • Oracle Dependency: Estimating expertise relies heavily on the quality of the initial outcome model. If this model is poor, the expertise estimate will be biased, potentially leading to incorrect model selection.
  • Real-World Validation: While the theoretical and semi-synthetic results are robust, validating this framework on purely real-world data—where counterfactuals are truly unobservable—remains a significant challenge.

Ultimately, this framework encourages practitioners to deeply consider why a treatment was assigned before blindly applying a causal inference algorithm.

Summary updated

Abstract

Decision-makers are often experts of their domain and take actions based on their domain knowledge. Doctors, for instance, may prescribe treatments by predicting the likely outcome of each available treatment. Actions of an expert thus naturally encode part of their domain knowledge, and can help make inferences within the same domain: Knowing doctors try to prescribe the best treatment for their patients, we can tell treatments prescribed more frequently are likely to be more effective. Yet in machine learning, the fact that most decision-makers are experts is often overlooked, and “expertise” is seldom leveraged as an inductive bias. This is especially true for the literature on treatment effect estimation, where often the only assumption made about actions is that of overlap. In this paper, we argue that expertise—particularly the type of expertise the decision-makers of a domain are likely to have—can be informative in designing and selecting methods for treatment effect estimation. We formally define two types of expertise, predictive and prognostic, and demonstrate empirically that: (i) the prominent type of expertise in a domain significantly influences the performance of different methods in treatment effect estimation, and (ii) it is possible to predict the type of expertise present in a dataset, which can provide a quantitative basis for model selection.
BibTeX
@inproceedings{huyuk2024defining,
  author = {Alihan Hüyük and Qiyao Wei and Alicia Curth and Mihaela van der Schaar},
  title = {{Defining Expertise: Applications to Treatment Effect Estimation}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=1YPfmglNRU},
}
Figure