Pareto Optimal Model Selection

50 concepts44 papers1 proposal

When building machine learning models, we are almost always trapped between two competing desires: we want models that are highly accurate, but we also want models that are simple, interpretable, and robust.

The traditional way to handle this trade-off is scalarization. We combine accuracy and complexity into a single score—for instance, by adding an $L_1$ or $L_2$ regularization penalty to our loss function. We choose a hyperparameter $\lambda$ to weight the penalty, run our optimization, and get a single model.

But this approach has a fundamental flaw: choosing $\lambda$ beforehand is blind guesswork. If we choose a value that is too high, our model underfits; if it is too low, the model suffers from structural "bloat," becoming an uninterpretable black box.

A powerful alternative paradigm treats model selection as a true multi-objective optimization problem. Instead of guessing a trade-off upfront, we optimize for accuracy and simplicity simultaneously as independent dimensions. This process maps out a Pareto frontier—a menu of optimal models where no single model can be made more accurate without also making it more complex. Only after seeing this entire frontier does the practitioner select the final model that best fits their specific domain constraints.


The Geometry of the Trade-Off

To understand why this works, imagine a two-dimensional space where the horizontal axis represents model complexity (e.g., the number of parameters or rules) and the vertical axis represents predictive error.

Error
  ^
  |  * (Model A: Simple but high error)
  |   \
  |    * (Model B: Balanced)
  |     \
  |      * (Model C: Highly accurate but complex)
  |--------------------------------------------> Complexity

In this space, a model dominates another if it is both less complex and less error-prone. The models that are not dominated by any others form the Pareto frontier.

As originally recognized in early work by Hisao Ishibuchi et al. in Selection of Linguistic Classification Rules Using Two-objective Genetic Algorithms, presenting users with this entire frontier completely changes the selection process. Instead of evaluating a single arbitrary model, practitioners can look at the slope of the frontier to quantify the exact "cost" of interpretability—seeing precisely how much accuracy they must sacrifice to gain a simpler model.


Why Evolutionary Algorithms?

Standard machine learning models are optimized using gradient descent, which requires smooth, differentiable loss functions. However, metrics for model complexity—such as the number of features, the count of logic rules, or the discrete depth of a mathematical expression tree—are inherently discrete and non-differentiable.

This makes Multi-Objective Evolutionary Algorithms (MOEAs), such as NSGA-II or GOMEA, uniquely suited for this task. Rather than tracking a single point in parameter space, evolutionary algorithms maintain a diverse population of candidate models. Over successive generations, the algorithm mutates, recombines, and selects models based on their proximity to the non-dominated frontier.

This approach is widely used across different model representations:


Expanding the Objectives

While the accuracy-complexity trade-off is the most common framing, the multi-objective approach easily scales to incorporate other critical model characteristics.

1. Interpretability Beyond Size

Complexity isn't just about counting parameters; it depends heavily on how humans perceive the model. As shown by Janek Thomas et al. in Multi-Objective Optimization of Performance and Interpretability of Tabular Supervised Machine Learning Models, we can introduce objectives for feature sparsity, interaction constraints, and monotonicity. This ensures that the resulting models conform to domain-specific logic (e.g., ensuring that a credit risk model's predicted risk only increases as a applicant's debt increases).

2. Stability and Robustness

In dynamical systems, a model might accurately predict the next time-step but fail catastrophically when simulated over long horizons due to accumulated errors. Dhruv Khandelwal et al. addressed this in Grammar-based Representation and Identification of Dynamical Systems by optimizing for two distinct objectives: short-term prediction accuracy and long-term simulation stability.

3. Expert Knowledge Alignment

When learning probabilistic structures like Bayesian networks, the data alone might suggest relationships that defy physical reality. In Learning Discretized Bayesian Networks with GOMEA, Damy M. F. Ha demonstrated that treating alignment with expert knowledge as an explicit third objective allows practitioners to find models that are both statistically sound and grounded in domain reality.


Making the Final Selection

Once the evolutionary algorithm has mapped out the Pareto frontier, how do we choose the single model to deploy?

The Validation Set Strategy

A common risk in multi-objective optimization is that the evolutionary process might overfit the Pareto front to the training data. To counter this, Christian Gagné et al. in Genetic Programming, Validation Sets, and Parsimony Pressure established a robust protocol: generate the Pareto-optimal set using the training data, but make the final selection by evaluating those specific frontier models on an independent validation set. This ensures that the chosen model generalizes well to unseen data.

Prioritizing Scientific Plausibility

In high-stakes fields like medicine or neuroscience, the "best" model is rarely the one with the absolute lowest training error. Seyed Mostafa Kia et al. demonstrated this principle in Interpretability of Multivariate Brain Maps in Linear Brain Decoding: Definition, and Heuristic Quantification in Multivariate Analysis of MEG Time-Locked Effects. By using a multi-objective criterion that balances generalization accuracy with interpretability metrics (like reproducibility and representativeness), the authors showed that accepting a marginal, fractional decrease in prediction accuracy can yield feature maps that are significantly more biologically plausible and scientifically useful.

By shifting from a single-objective mindset to Pareto-optimal selection, machine learning moves away from rigid, automated assumptions about regularization and toward a transparent, flexible framework where human experts can make informed trade-offs.

Key Foundational Work

Selection of Linguistic Classfication Rules Using Two-objective Genetic Algorithms (1995) — contributed 1 concept elements to this direction.
Approximate Theory Formation: An Explanation-Based Approach (1988) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_multi_objective_model_selection_84d8b286,
  title = {multi-objective model selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multi-objective-model-selection/rdr_84d8b286563a4f5ab5abe947adcfe3dc},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_multi_objective_model_selection_84d8b286_v1,
  title = {multi-objective model selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multi-objective-model-selection/rdr_84d8b286563a4f5ab5abe947adcfe3dc/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake