MINTY: Rule-based models that minimize the need for imputing features with missing values

Lena Stempfle, Fredrik D. Johansson

Abstract

Rule models are often preferred in prediction tasks with tabular inputs as they can be easily interpreted using natural language and provide predictive performance on par with more complex models. However, most rule models’ predictions are undefined or ambiguous when some inputs are missing, forcing users to rely on statistical imputation models or heuristics like zero imputation, undermining the interpretability of the models. In this work, we propose fitting concise yet precise rule models that learn to avoid relying on features with missing values and, therefore, limit their reliance on imputation at test time. We develop MINTY, a method that learns rules in the form of disjunctions between variables that act as replacements for each other when one or more is missing. This results in a sparse linear rule model, regularized to have small dependence on features with missing values, that allows a trade-off between goodness of fit, interpretability, and robustness to missing values at test time. We demonstrate the value of MINTY in experiments using synthetic and real-world data sets and find its predictive performance comparable or favorable to baselines, with smaller reliance on features with missing values.

Introduction

In high-stakes domains like medicine and finance, interpretability is paramount. Rule-based models, such as "If Age > 65 AND Blood Pressure > 140, then Risk is High," are favored because they align with human reasoning. However, these models are notoriously brittle when faced with missing data. If a single feature in a rule is missing, the entire rule becomes undefined.

The conventional solution is data imputation—using secondary models to estimate missing values. This introduces a "black-box" element into an otherwise transparent process, potentially masking biases or errors. MINTY (Minimize the Need for Testing/Imputation) addresses this by building models that are inherently robust to missingness, seeking to bypass the need for imputation altogether.

Methodology: Robustness through Redundancy

MINTY builds upon Generalized Linear Rule Models (GLRM) but introduces a critical innovation: disjunctive rules (OR-clauses). Instead of strict conjunctions (AND), MINTY searches for features that can serve as functional replacements for one another.

For instance, a clinical rule might be formulated as: (Memory Test Score < 24) OR (Prior Alzheimer's Diagnosis). If the memory test result is missing, the model can still evaluate the rule based on the presence of a prior diagnosis. This redundancy creates a "fail-safe" mechanism within the model's logic.

Example of a MINTY scorecard using disjunctive rules to handle missing data.

To optimize these models, MINTY employs two main strategies:

  1. Column Generation: An efficient algorithm to navigate the vast space of possible OR-combinations to identify the most predictive features.
  2. Missingness Regularization: A tunable penalty parameter ($\gamma$) that discourages the model from relying on features that are frequently missing. This allows practitioners to explicitly trade off predictive accuracy for increased robustness.

Experimental Evaluation

The authors tested MINTY across several datasets:

  • ADNI: A real-world medical dataset for Alzheimer’s progression with natural missingness.
  • Life Expectancy & Housing: Datasets where missingness was manually introduced to test robustness.
  • Synthetic Data: Used to verify the algorithm's ability to identify truly redundant variables.

Key Results and Insights

MINTY demonstrates that redundancy, often viewed as a weakness in machine learning, can be a significant strength for robustness. By intentionally seeking out "replacement" variables, MINTY maintains high performance even when data is incomplete.

Performance and Trade-offs

On the ADNI and Housing datasets, MINTY achieved $R^2$ scores comparable to sophisticated black-box models like XGBoost. Crucially, it did so while drastically reducing "missingness reliance"—the frequency with which a prediction requires an imputed value. In the ADNI task, reliance dropped from over 50% in standard models to nearly 0% with minimal impact on accuracy.

Trade-off between predictive performance ($R^2$) and missingness reliance.

Avoiding the "Missingness Bias"

Unlike models that treat the absence of data as a predictive feature in itself (which can be risky if data collection policies change), MINTY focuses on the underlying signal. This ensures the model remains valid even if the patterns of missingness shift over time.

Limitations and Future Directions

Despite its advantages, MINTY faces some challenges:

  • Complexity: Finding optimal "OR" rules is computationally intensive (NP-hard), necessitating the use of heuristics like beam search.
  • Linearity and Discretization: As a linear combination of rules based on binned continuous data, MINTY may lack the precision of deep neural networks for highly non-linear problems.

Future research aims to apply this "imputation-free" philosophy to other interpretable structures, such as decision trees, to further enhance the reliability of human-centric AI.

Summary updated

Abstract

Rule models are often preferred in prediction tasks with tabular inputs as they can be easily interpreted using natural language and provide predictive performance on par with more complex models. However, most rule models’ predictions are undefined or ambiguous when some inputs are missing, forcing users to rely on statistical imputation models or heuristics like zero imputation, undermining the interpretability of the models. In this work, we propose fitting concise yet precise rule models that learn to avoid relying on features with missing values and, therefore, limit their reliance on imputation at test time. We develop MINTY, a method that learns rules in the form of disjunctions between variables that act as replacements for each other when one or more is missing. This results in a sparse linear rule model, regularized to have small dependence on features with missing values, that allows a trade-off between goodness of fit, interpretability, and robustness to missing values at test time. We demonstrate the value of MINTY in experiments using synthetic and real-world data sets and find its predictive performance comparable or favorable to baselines, with smaller reliance on features with missing values.
BibTeX
@inproceedings{stempfle2024minty,
  author = {Lena Stempfle and Fredrik D. Johansson},
  title = {{MINTY: Rule-based models that minimize the need for imputing features with missing values}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2024},
  url = {https://openreview.net/forum?id=Lm0D8Bcrvw},
}
Figure