Self-paced Mixture of Regressions

Longfei Han, Dingwen Zhang, Dong Huang, Xiaojun Chang, Jun Ren, Senlin Luo, Junwei Han

Abstract

Self-paced Mixture of Regressions (SPMoR) is a robust framework designed to handle heterogeneous and noisy data by mimicking human learning patterns. By integrating a self-paced regularizer based on the Exclusive LASSO, the model effectively filters out intra-component outliers and balances learning across unevenly distributed data groups. Experimental results on synthetic data, age estimation (MORPH), and glucose level prediction (NHANES) demonstrate that SPMoR significantly outperforms traditional Mixture of Regressions and other state-of-the-art robust regression techniques.

Introduction

In real-world applications, regression data is rarely clean or uniform. For instance, predicting a person's age from a photograph involves data that is inherently heterogeneous—aging patterns vary significantly across different demographics—and often noisy, containing outliers that do not follow the general trend.

Standard Mixture of Regressions (MoR) models attempt to handle heterogeneity by employing a "divide and conquer" strategy with local experts. However, they suffer from two primary weaknesses:

  1. Intra-component outliers: A small number of noisy data points within a specific group can heavily bias that group's expert.
  2. Inter-component imbalance: Models naturally favor majority groups, leading to poor performance on minority groups.
Figure 1: Conceptual overview of SPMoR. The model identifies local experts for different data groups while effectively ignoring outliers (marked with 'x') and selecting high-confidence samples.

The SPMoR Framework

To address these limitations, the authors introduced SPMoR, which incorporates Self-paced Learning (SPL) into the MoR framework. SPL follows a curriculum-based approach: the model initially trains on "easy" samples (those with low prediction error) and gradually includes more complex or noisy samples as its proficiency increases.

The Role of Exclusive LASSO

The technical core of SPMoR is the use of the Exclusive LASSO ($l_{1,2}$ norm) as a regularizer. This mathematical constraint serves a dual purpose:

  • Intra-group Sparsity: It forces the model to be selective within each component, effectively discarding outliers that do not fit the local regression line.
  • Inter-group Non-sparsity: It prevents the model from completely ignoring smaller groups. By encouraging the selection of at least some samples from every group, it ensures that minority experts are still trained on confident data points, maintaining a balance across the entire dataset.
Figure 3: The self-paced learning process over iterations. The model starts with a broad initialization and gradually refines its selection, focusing on reliable samples in both components while excluding outliers at the bottom.

Experimental Results

The authors validated SPMoR across three distinct domains, showing consistent improvements in both robustness and accuracy.

Synthetic Simulations

In controlled tests with 5% outliers, standard MoR models saw their Mean Squared Error (MSE) balloon to 0.008055. In contrast, SPMoR maintained an MSE of 0.000175—a nearly 46-fold improvement in precision under noisy conditions.

Age and Glucose Estimation

  • MORPH Dataset (Age Estimation): On a massive dataset of over 55,000 face images, a hierarchical version of the model (SPMoR+) achieved a Mean Absolute Error (MAE) of 3.55 years, surpassing previous benchmarks like HSVR (3.60) and LSVR (4.31).
  • NHANES Dataset (Glucose Estimation): For predicting HbA1c levels (critical for diabetes diagnosis), SPMoR+ achieved an MSE of 0.279, significantly outperforming Support Vector Regression (0.510) and standard MoE (0.349).

Conclusion

SPMoR represents a significant step forward in robust regression. By combining the group-wise flexibility of MoR with the noise-filtering capabilities of self-paced learning, it provides a principled way to handle the messiness of real-world data. Future work may explore "soft" weighting schemes to further refine sample selection and apply this curriculum-based logic to other complex computer vision tasks like object tracking.

Summary updated

Abstract

Mixture of regressions (MoR) is the well-established and effective approach to model discontinuous and heterogeneous data in regression problems. Existing MoR approaches assume smooth joint distribution for its good anlaytic properties. However, such assumption makes existing MoR very sensitive to intra-component outliers (the noisy training data residing in certain components) and the inter-component imbalance (the different amounts of training data in different components). In this paper, we make the earliest effort on Self-paced Learning (SPL) in MoR, i.e., Self-paced mixture of regressions (SPMoR) model. We propose a novel self-paced regularizer based on the Exclusive LASSO, which improves inter-component balance of training data. As a robust learning regime, SPL pursues confidence sample reasoning. To demonstrate the effectiveness of SPMoR, we conducted experiments on both the sythetic examples and real-world applications to age estimation and glucose estimation. The results show that SPMoR outperforms the state-of-the-arts methods.
BibTeX
@inproceedings{han2017self,
  author = {Longfei Han and Dingwen Zhang and Dong Huang and Xiaojun Chang and Jun Ren and Senlin Luo and Junwei Han},
  title = {{Self-paced Mixture of Regressions}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.24963/ijcai.2017/252},
}
Figure