Abstract
Decision-makers are often experts of their domain and take actions based on their domain knowledge. Doctors, for instance, may prescribe treatments by predicting the likely outcome of each available treatment. Actions of an expert thus naturally encode part of their domain knowledge, and can help make inferences within the same domain: Knowing doctors try to prescribe the best treatment for their patients, we can tell treatments prescribed more frequently are likely to be more effective. Yet in machine learning, the fact that most decision-makers are experts is often overlooked, and "expertise" is seldom leveraged as an inductive bias. This is especially true for the literature on treatment effect estimation, where often the only assumption made about actions is that of overlap. In this paper, we argue that expertise - particularly the type of expertise the decision-makers of a domain are likely to have - can be informative in designing and selecting methods for treatment effect estimation. We formally define two types of expertise, predictive and prognostic, and demonstrate empirically that: (i) the prominent type of expertise in a domain significantly influences the performance of different methods in treatment effect estimation, and (ii) it is possible to predict the type of expertise present in a dataset, which can provide a quantitative basis for model selection.
Introduction
In the field of Causal Inference, human decision-makers—such as doctors, educators, or policy-makers—are often viewed as a source of statistical noise. Their "expertise" creates confounding: because they do not assign treatments randomly, the distribution of features (like age or health markers) differs significantly between treated and untreated groups. This covariate shift makes it difficult to isolate the true effect of a treatment.
Research presented at ICLR 2024 suggests a paradigm shift: instead of attempting to "balance away" the influence of experts, we should leverage their expertise as a powerful inductive bias. By formalizing what it means to be an expert, we can better select machine learning models that align with the underlying logic of the data-generating process.
Defining Two Types of Expertise
The authors introduce two formal, entropy-based definitions of expertise to categorize how decisions are made in a dataset:
- Prognostic Expertise ($E_{prog}$): This occurs when actions are informed by potential outcomes in general. For example, a teacher might provide additional tutoring to students who are struggling overall, regardless of whether those specific students are the most likely to show the highest improvement from that specific intervention.
- Predictive Expertise ($E_{pred}$): This occurs when actions are informed specifically by the treatment effect—the difference between potential outcomes. A doctor exhibits predictive expertise when they prescribe a specific medication only to patients for whom that drug is expected to work significantly better than the alternative.
The Balancing Paradox
Most modern Treatment Effect Estimation (TEE) methods rely on "balancing representations." These models, such as CFRNet, attempt to force the machine learning model to ignore the features the expert used to make their decision, effectively trying to make the observational data look like a randomized controlled trial.
However, the authors identify a "Balancing Paradox": when predictive expertise is high, balancing is actually detrimental. By trying to make treatment assignment look random, these models delete the very signal that the expert intentionally placed in the data. If a doctor is a predictive expert, the fact that they chose a treatment is the most valuable piece of information available. In these cases, Action-predictive models (like DragonNet) that incorporate the treatment assignment logic tend to perform much better.
The Expertise-Informed Pipeline
Since the specific type of expertise in a real-world dataset is rarely known, the authors developed a quantitative pipeline to estimate it:
- Estimation: Using an action-predictive model, the system calculates the ratio of predictive to prognostic expertise.
- Model Selection: If the predictive ratio is high (>0.5), the pipeline selects an Action-predictive model. If the ratio is low, it defaults to a Balancing model.
- Trade-off: The paper proves via Proposition 1 that there is a fundamental trade-off between expertise and "overlap." While high expertise makes traditional statistical overlap harder to achieve, it provides more information that expertise-aware machine learning can exploit.
Empirical Results
The framework was validated using synthetic environments based on real-world data, including TCGA (The Cancer Genome Atlas) and the News dataset. The performance was measured using PEHE (Precision in Estimation of Heterogeneous Effects), where lower scores indicate better accuracy.
- Predictive Scenarios: Action-predictive models significantly outperformed balancing models as expertise levels increased.
- Prognostic Scenarios: Balancing models remained superior, as they filtered out prognostic variables that did not impact the treatment delta.
- Overall Performance: The "Expertise-informed" selector achieved the best results across the board. In the TCGA environment, it reached an average PEHE of 1.096, outperforming standard Baselines (1.134) and Balancing models (1.188).
Conclusion and Future Directions
This research demonstrates that expertise is not just a source of confounding, but a usable signal. By identifying whether a decision-maker is acting on prognostic or predictive knowledge, researchers can make more informed choices about which CATE models to deploy.
A primary limitation remains the "No Hidden Confounding" assumption—the theory assumes the researcher can see the same data the expert used. Future work may extend these definitions to Inverse Decision Modeling, potentially allowing AI to "reverse-engineer" the logic of human experts to improve automated decision systems.
Summary updated