TL;DR
Causal features are often touted as the "holy grail" for building machine learning models that generalize to new environments. However, this large-scale empirical study of 16 tabular datasets finds that "kitchen-sink" models using all available features consistently outperform causal predictors in both in-domain and out-of-domain accuracy.
What Problem Does This Paper Address?
Machine learning models are notoriously brittle: a model trained on data from one hospital or geographic region often sees its accuracy plummet when deployed in a new one. This is known as the "domain generalization" problem.
A popular theory in the research community suggests that causal relationships are the key to solving this. The logic is that while correlations might change between environments (e.g., a specific hospital's coding practices), the underlying causal mechanisms (e.g., how a disease progresses) should remain invariant. Therefore, a model that only looks at "causal features" should, in theory, be more robust to distribution shifts. This paper puts that theory to a rigorous empirical test across a wide range of real-world tabular datasets.
Methods
The authors evaluated 16 prediction tasks using a three-pronged approach:
- Feature Categorization: For each task, they manually categorized features based on expert knowledge and literature into four groups:
- Causal: Features that almost certainly have a direct causal influence on the target.
- Arguably Causal: A more inclusive set including plausible causes or variables with bidirectional relationships.
- Anti-causal: Features that are likely effects of the target variable.
- All: Every available feature in the dataset.
- Algorithm Comparison: They tested "standard" models (like XGBoost, LightGBM, and various Neural Networks) against specialized Domain Generalization methods (like Group DRO) and Causal ML methods (like Invariant Risk Minimization (IRM) and Risk Extrapolation (REx)).
- Evaluation Metrics: They measured both absolute accuracy (in-domain and out-of-domain) and the "Shift Gap"—the absolute drop in performance when moving from the training domain to a new test domain.
Datasets
The study utilized 16 classification tasks derived from established benchmarks like TableShift. These datasets cover diverse domains with natural distribution shifts:
- Health: Diabetes and hypertension diagnosis (BRFSS), hospital mortality and ICU stay length (MIMIC-III), and sepsis onset (PhysioNet).
- Economics/Social: Income level, unemployment, and food stamp recipiency (ACS/Folktables), and poverty levels (SIPP).
- Politics/Education: Voting behavior (ANES), college completion rates (College Scorecard), and student performance (ASSISTments).
Key Insights and Ideas
- The "All Features" Dominance: In all 16 tasks, models using all available features achieved the highest out-of-domain accuracy. Restricting a model to causal features actually made it worse at generalizing, not better.
- The Anti-Causal Paradox: Theoretically, including "anti-causal" features (effects of the target) should make a model vulnerable to shifts. Surprisingly, the authors found that adding anti-causal features to the causal set actually improved out-of-domain performance in every case where they were available.
- Causal ML vs. Standard ML: Specialized causal algorithms like IRM and REx did not provide a "silver bullet." Their performance generally fell somewhere between the conservative causal models and the inclusive "arguably causal" models, never surpassing the standard models trained on all features.
- Causal Discovery Limitations: Automated causal discovery algorithms (like PC or ICP) often failed to run on these datasets or selected feature sets that performed poorly compared to human-selected "arguably causal" sets.
Key Results
- Accuracy: Models trained on all features Pareto-dominated causal selections in 16/16 tasks for both in-domain and out-of-domain accuracy.
- Shift Gap: The absolute drop in accuracy (the shift gap) was actually smaller for models using all features in 7 out of 16 tasks. In the remaining tasks, the gaps were statistically indistinguishable.
- Predictive Power: Causal models were often so restricted that they had very low predictive power, sometimes performing no better than a "constant" predictor that simply guesses the majority label.
Limitations and Future Work
The authors acknowledge several important caveats:
- Tabular Data Specifics: These results are specific to tabular data. Causal methods have shown more promise in other modalities, such as computer vision (e.g., identifying animals in different backgrounds).
- The "Bayes Optimal" Explanation: The authors suggest a theoretical reason for their findings: if domains are simply "positive reweightings" of each other, the best model for one domain is mathematically the best for all. In many tabular settings, the distribution shifts might not be "strong" enough to favor causal invariance over raw predictive power.
- Future Benchmarks: The study suggests that the "burden of proof" is now on proponents of causal ML to provide real-world benchmark datasets where these methods actually provide a tangible benefit over standard empirical risk minimization.
In short: if you're working with tabular data and want to generalize to a new domain, your best bet is likely to train the strongest possible model on all the data you have, rather than trying to prune features based on causal intuition.
Summary updated