Abstract
Self-paced Mixture of Regressions (SPMoR) is a robust framework designed to handle heterogeneous and noisy data by mimicking human learning patterns. By integrating a self-paced regularizer based on the Exclusive LASSO, the model effectively filters out intra-component outliers and balances learning across unevenly distributed data groups. Experimental results on synthetic data, age estimation (MORPH), and glucose level prediction (NHANES) demonstrate that SPMoR significantly outperforms traditional Mixture of Regressions and other state-of-the-art robust regression techniques.
Introduction
In real-world applications, regression data is rarely clean or uniform. For instance, predicting a person's age from a photograph involves data that is inherently heterogeneous—aging patterns vary significantly across different demographics—and often noisy, containing outliers that do not follow the general trend.
Standard Mixture of Regressions (MoR) models attempt to handle heterogeneity by employing a "divide and conquer" strategy with local experts. However, they suffer from two primary weaknesses:
- Intra-component outliers: A small number of noisy data points within a specific group can heavily bias that group's expert.
- Inter-component imbalance: Models naturally favor majority groups, leading to poor performance on minority groups.
The SPMoR Framework
To address these limitations, the authors introduced SPMoR, which incorporates Self-paced Learning (SPL) into the MoR framework. SPL follows a curriculum-based approach: the model initially trains on "easy" samples (those with low prediction error) and gradually includes more complex or noisy samples as its proficiency increases.
The Role of Exclusive LASSO
The technical core of SPMoR is the use of the Exclusive LASSO ($l_{1,2}$ norm) as a regularizer. This mathematical constraint serves a dual purpose:
- Intra-group Sparsity: It forces the model to be selective within each component, effectively discarding outliers that do not fit the local regression line.
- Inter-group Non-sparsity: It prevents the model from completely ignoring smaller groups. By encouraging the selection of at least some samples from every group, it ensures that minority experts are still trained on confident data points, maintaining a balance across the entire dataset.
Experimental Results
The authors validated SPMoR across three distinct domains, showing consistent improvements in both robustness and accuracy.
Synthetic Simulations
In controlled tests with 5% outliers, standard MoR models saw their Mean Squared Error (MSE) balloon to 0.008055. In contrast, SPMoR maintained an MSE of 0.000175—a nearly 46-fold improvement in precision under noisy conditions.
Age and Glucose Estimation
- MORPH Dataset (Age Estimation): On a massive dataset of over 55,000 face images, a hierarchical version of the model (SPMoR+) achieved a Mean Absolute Error (MAE) of 3.55 years, surpassing previous benchmarks like HSVR (3.60) and LSVR (4.31).
- NHANES Dataset (Glucose Estimation): For predicting HbA1c levels (critical for diabetes diagnosis), SPMoR+ achieved an MSE of 0.279, significantly outperforming Support Vector Regression (0.510) and standard MoE (0.349).
Conclusion
SPMoR represents a significant step forward in robust regression. By combining the group-wise flexibility of MoR with the noise-filtering capabilities of self-paced learning, it provides a principled way to handle the messiness of real-world data. Future work may explore "soft" weighting schemes to further refine sample selection and apply this curriculum-based logic to other complex computer vision tasks like object tracking.
Summary updated