When building machine learning models, we are almost always trapped between two competing desires: we want models that are highly accurate, but we also want models that are simple, interpretable, and robust.
The traditional way to handle this trade-off is scalarization. We combine accuracy and complexity into a single score—for instance, by adding an $L_1$ or $L_2$ regularization penalty to our loss function. We choose a hyperparameter $\lambda$ to weight the penalty, run our optimization, and get a single model.
But this approach has a fundamental flaw: choosing $\lambda$ beforehand is blind guesswork. If we choose a value that is too high, our model underfits; if it is too low, the model suffers from structural "bloat," becoming an uninterpretable black box.
A powerful alternative paradigm treats model selection as a true multi-objective optimization problem. Instead of guessing a trade-off upfront, we optimize for accuracy and simplicity simultaneously as independent dimensions. This process maps out a Pareto frontier—a menu of optimal models where no single model can be made more accurate without also making it more complex. Only after seeing this entire frontier does the practitioner select the final model that best fits their specific domain constraints.
The Geometry of the Trade-Off
To understand why this works, imagine a two-dimensional space where the horizontal axis represents model complexity (e.g., the number of parameters or rules) and the vertical axis represents predictive error.
Error
^
| * (Model A: Simple but high error)
| \
| * (Model B: Balanced)
| \
| * (Model C: Highly accurate but complex)
|--------------------------------------------> Complexity
In this space, a model dominates another if it is both less complex and less error-prone. The models that are not dominated by any others form the Pareto frontier.
As originally recognized in early work by Hisao Ishibuchi et al. in Selection of Linguistic Classification Rules Using Two-objective Genetic Algorithms, presenting users with this entire frontier completely changes the selection process. Instead of evaluating a single arbitrary model, practitioners can look at the slope of the frontier to quantify the exact "cost" of interpretability—seeing precisely how much accuracy they must sacrifice to gain a simpler model.
Why Evolutionary Algorithms?
Standard machine learning models are optimized using gradient descent, which requires smooth, differentiable loss functions. However, metrics for model complexity—such as the number of features, the count of logic rules, or the discrete depth of a mathematical expression tree—are inherently discrete and non-differentiable.
This makes Multi-Objective Evolutionary Algorithms (MOEAs), such as NSGA-II or GOMEA, uniquely suited for this task. Rather than tracking a single point in parameter space, evolutionary algorithms maintain a diverse population of candidate models. Over successive generations, the algorithm mutates, recombines, and selects models based on their proximity to the non-dominated frontier.
This approach is widely used across different model representations:
- Symbolic Regression: In tools like PySR, reviewed by Alberto Tonda in Review of PySR: high-performance symbolic regression in Python and Julia, evolutionary search tracks a Pareto front of mathematical equations. This allows algorithms to combat "bloat"—the tendency of symbolic models to grow exponentially large without meaningful gains in accuracy, a challenge detailed by Michael Kommenda et al. in Complexity Measures for Multi-objective Symbolic Regression.
- Tree-Based Networks: William G. La Cava et al. introduced FEAT in Learning concise representations for regression by evolving networks of trees, which uses multi-objective evolution to grow small, interpretable networks of trees that match the performance of massive black-box models like XGBoost.
- Rule-Based Classifiers: For multi-label classification, the MINOTAUR framework developed by Thiago Zafalon Miranda et al. in Generation of consistent sets of multi-label classification rules with a multi-objective evolutionary algorithm navigates the trade-off by evolving populations of models to minimize the total number of classification rules while maximizing accuracy.
Expanding the Objectives
While the accuracy-complexity trade-off is the most common framing, the multi-objective approach easily scales to incorporate other critical model characteristics.
1. Interpretability Beyond Size
Complexity isn't just about counting parameters; it depends heavily on how humans perceive the model. As shown by Janek Thomas et al. in Multi-Objective Optimization of Performance and Interpretability of Tabular Supervised Machine Learning Models, we can introduce objectives for feature sparsity, interaction constraints, and monotonicity. This ensures that the resulting models conform to domain-specific logic (e.g., ensuring that a credit risk model's predicted risk only increases as a applicant's debt increases).
2. Stability and Robustness
In dynamical systems, a model might accurately predict the next time-step but fail catastrophically when simulated over long horizons due to accumulated errors. Dhruv Khandelwal et al. addressed this in Grammar-based Representation and Identification of Dynamical Systems by optimizing for two distinct objectives: short-term prediction accuracy and long-term simulation stability.
3. Expert Knowledge Alignment
When learning probabilistic structures like Bayesian networks, the data alone might suggest relationships that defy physical reality. In Learning Discretized Bayesian Networks with GOMEA, Damy M. F. Ha demonstrated that treating alignment with expert knowledge as an explicit third objective allows practitioners to find models that are both statistically sound and grounded in domain reality.
Making the Final Selection
Once the evolutionary algorithm has mapped out the Pareto frontier, how do we choose the single model to deploy?
The Validation Set Strategy
A common risk in multi-objective optimization is that the evolutionary process might overfit the Pareto front to the training data. To counter this, Christian Gagné et al. in Genetic Programming, Validation Sets, and Parsimony Pressure established a robust protocol: generate the Pareto-optimal set using the training data, but make the final selection by evaluating those specific frontier models on an independent validation set. This ensures that the chosen model generalizes well to unseen data.
Prioritizing Scientific Plausibility
In high-stakes fields like medicine or neuroscience, the "best" model is rarely the one with the absolute lowest training error. Seyed Mostafa Kia et al. demonstrated this principle in Interpretability of Multivariate Brain Maps in Linear Brain Decoding: Definition, and Heuristic Quantification in Multivariate Analysis of MEG Time-Locked Effects. By using a multi-objective criterion that balances generalization accuracy with interpretability metrics (like reproducibility and representativeness), the authors showed that accepting a marginal, fractional decrease in prediction accuracy can yield feature maps that are significantly more biologically plausible and scientifically useful.
By shifting from a single-objective mindset to Pareto-optimal selection, machine learning moves away from rigid, automated assumptions about regularization and toward a transparent, flexible framework where human experts can make informed trade-offs.
Key Foundational Work
Selection of Linguistic Classfication Rules Using Two-objective Genetic Algorithms (1995) — contributed 1 concept elements to this direction.
Approximate Theory Formation: An Explanation-Based Approach (1988) — contributed 1 concept elements to this direction.