MINTY: Rule-based Models that Minimize the Need for Imputing Features with Missing Values

Lena Stempfle, Fredrik D. Johansson

Abstract

Rule models are often preferred in prediction tasks with tabular inputs as they can be easily interpreted using natural language and provide predictive performance on par with more complex models. However, most rule models' predictions are undefined or ambiguous when some inputs are missing, forcing users to rely on statistical imputation models or heuristics like zero imputation, undermining the interpretability of the models. In this work, we propose fitting concise yet precise rule models that learn to avoid relying on features with missing values and, therefore, limit their reliance on imputation at test time. We develop MINTY, a method that learns rules in the form of disjunctions between variables that act as replacements for each other when one or more is missing. This results in a sparse linear rule model, regularized to have small dependence on features with missing values, that allows a trade-off between goodness of fit, interpretability, and robustness to missing values at test time. We demonstrate the value of MINTY in experiments using synthetic and real-world data sets and find its predictive performance comparable or favorable to baselines, with smaller reliance on features with missing values.

Introduction

In real-world tabular datasets—especially in healthcare or social sciences—it is common for some features to be missing at the time of prediction. Standard approaches usually involve "imputation," where a secondary model guesses the missing values. However, imputation has two major drawbacks: it can introduce statistical bias, and it undermines the interpretability of simple rule-based models. If a clinician uses a rule like "If Age > 70, then Risk is High," but the patient's age is missing, an imputed age makes the final decision feel like a "black box."

Existing "missing-aware" models often use "fallback" rules or missingness indicators, but these can become overly complex. MINTY aims to create a model that is inherently robust to missingness by design, preferring rules that can be evaluated using whatever data is actually available.

Methods

The core of MINTY is the Generalized Linear Rule Model (GLRM), but with a twist. Instead of rules based on single features, MINTY learns disjunctive rules—logical "OR" statements. For example, a rule might be: (Protein A > 10) OR (Protein B > 5). If Protein A is missing but Protein B is observed and true, the rule is satisfied. This exploits the fact that many variables in a dataset are correlated and can act as "replacements" for one another.

Example of a MINTY rule model for a patient named Anna. The model uses disjunctive rules to handle missing features (PTAU, MAR.) by relying on observed redundant features (TAU).

Key technical components include:

  • Reliance Regularization: A tunable penalty parameter ($\gamma$) that specifically penalizes rules that frequently cannot be determined due to missing values.
  • Column Generation: Since the number of possible "OR" combinations is exponentially large, MINTY uses a "column generation" strategy to iteratively add only the most useful rules to the model.
  • Sparse Linear Combination: The final model is a sparse linear combination of these disjunctive rules, making it as easy to read as a medical risk score.

Results and Discussion

The authors evaluated MINTY on both synthetic and real-world regression tasks, including the ADNI (Alzheimer’s Disease Neuroimaging Initiative) dataset for cognitive decline, WHO life expectancy data, and Ames housing prices. MINTY achieved the best (ADNI, Housing) or second-best (Life) predictive performance ($R^2$) compared to baselines like LASSO with MICE imputation, Decision Trees, and XGBoost.

Extracted figure from page 7 of MINTY: Rule-based Models that Minimize the Need for Imputing Features with Missing Values

A key insight is the Interpretability-Robustness Trade-off. By adjusting the $\gamma$ parameter, users can choose between a model that is slightly more accurate (but relies more on imputation) and one that is slightly less accurate but can be evaluated for 100% of patients without any "guessing." On the ADNI dataset, MINTY could be tuned to have zero reliance ($\bar{ ho} = 0$) on missing values while still outperforming XGBoost, which had to rely on "default" directions for over 50% of its predictions.

Conclusion

MINTY treats feature correlation not as a problem to be removed, but as a source of robustness. By grouping correlated features into "OR" clauses, the model builds in its own "backup" system. While MINTY requires some level of feature redundancy to be effective, it offers a powerful framework for creating interpretable models that remain functional in the face of incomplete data. Future work may extend this "reliance regularization" concept to other model classes, such as decision trees or random forests.

Summary updated

Abstract

Rule models are often preferred in prediction tasks with tabular inputs as they can be easily interpreted using natural language and provide predictive performance on par with more complex models. However, most rule models' predictions are undefined or ambiguous when some inputs are missing, forcing users to rely on statistical imputation models or heuristics like zero imputation, undermining the interpretability of the models. In this work, we propose fitting concise yet precise rule models that learn to avoid relying on features with missing values and, therefore, limit their reliance on imputation at test time. We develop MINTY, a method that learns rules in the form of disjunctions between variables that act as replacements for each other when one or more is missing. This results in a sparse linear rule model, regularized to have small dependence on features with missing values, that allows a trade-off between goodness of fit, interpretability, and robustness to missing values at test time. We demonstrate the value of MINTY in experiments using synthetic and real-world data sets and find its predictive performance comparable or favorable to baselines, with smaller reliance on features with missing values.
BibTeX
@inproceedings{stempfle2024minty,
  author = {Lena Stempfle and Fredrik D. Johansson},
  title = {{MINTY: Rule-based Models that Minimize the Need for Imputing Features with Missing Values}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2024},
  url = {https://openreview.net/forum?id=Lm0D8Bcrvw},
}
Figure