In the pursuit of higher accuracy, machine learning research often gravitates toward increasingly complex "black-box" architectures like deep neural networks and ensemble methods. However, a growing body of evidence across medicine, manufacturing, and social science suggests that for many high-stakes tasks, the "complexity tax" is not worth paying.
Researchers are finding that simple, regularized linear models—such as Logistic Regression and Lasso—frequently match or even exceed the performance of their more complex counterparts. The intuition is simple: in many real-world domains, the most critical signals are additive and stable. When these signals are captured through careful feature engineering, the marginal gains of non-linear modeling vanish, leaving only the disadvantages of opacity and instability.
The Performance Parity Phenomenon
The most striking finding across these studies is the lack of a significant "performance gap" between simple and complex models. In A Comparison of Supervised Machine Learning Algorithms and Feature Vectors for MS Lesion Segmentation Using Multimodal Structural MRI, Elizabeth Sweeney and colleagues demonstrated that for MRI segmentation, the quality of the feature vector (incorporating spatial context) mattered far more than the choice of algorithm. Once robust features were present, simple logistic regression performed as well as computationally intensive ensembles.
This pattern repeats in diverse fields:
- Autism Diagnosis: Sébastien Lévy and Dennis P. Wall showed that a sparse pipeline could reduce a 30-behavior exam to just 5–10 features using linear models without losing diagnostic accuracy in Sparsifying machine learning models identify stable subsets of predictive features for behavioral detection of autism.
- Public Health: In predicting COVID-19 severity, Saeed Shakibfar found that simple linear models achieved risk stratification comparable to complex ensembles in Machine learning-driven development of a disease risk score for COVID-19 hospitalization and mortality.
- Linguistics: Even in lexical selection, interpretable linear models were found to be nearly competitive with large-scale transformer models while providing human-readable rules for language learners.
Transparency as a Functional Requirement
In high-stakes environments, interpretability is not a "nice-to-have" feature; it is a requirement for safety, trust, and validation. Ioannis Patras and Niki Maria Foteinopoulou argue in their review of schizophrenia symptom estimation that linear architectures like Support Vector Machines are preferred because they maintain a direct, verifiable mapping between biological markers and clinical predictions.
This transparency allows for "glass box" deployment. For example, Yochai Edlitz developed a five-blood-test model for diabetes that uses logistic regression to create simple scoring systems. These systems allow clinicians to manually calculate risk at the bedside, fostering a level of trust that a "black box" cannot provide. Similarly, Stephen Wai Hang Kwok distilled 92 variables into a 16-point bedside tool for COVID-19, prioritizing ease of implementation over marginal accuracy gains.
Feature Engineering Over Algorithmic Complexity
The success of simpler models often stems from the integration of domain expertise into the feature engineering process. In Anomaly detection for yield improvement in glass production, Haruo Yonemori combined Lasso regression with Fault Tree Analysis. By grouping sensor data based on physical domain knowledge, the team created a system that engineers could actually trust and act upon, which is often more valuable than a slightly more accurate but inexplicable alert.
This philosophy is echoed in AI-augmented histopathologic review by Martin C. Stumpe, where linear models were used to ground AI recommendations in observable physical features like cell density. When a practitioner can see why a model reached a conclusion—because it aligns with their own professional observations—the AI becomes a collaborator rather than a replacement.
Hybrid Approaches to Complexity
When the underlying data is truly heterogeneous, researchers are turning to hybrid structures that preserve the benefits of linearity. Chirag Nagpal and colleagues proposed a Mixture of Experts (MoE) approach for survival analysis. Instead of one deep network, they used a gating function to coordinate a set of simple linear models. This outperformed "black-box" models while remaining interpretable, as clinicians could identify which specific features drove risk for different patient subgroups.
Ultimately, as Γεώργιος Παπουτσόγλου notes in Machine learning approaches in microbiome research, the marginal gains of complex models often do not justify the loss of transparency. By prioritizing parsimony and stability, researchers are building tools that are not just predictive, but actionable in the real world.