Learning Sample-Specific Models with Low-Rank Personalized Regression

Benjamin J. Lengerich, Bryon Aragam, Eric P. Xing

TL;DR

This paper introduces "Personalized Regression" (PR), a novel framework for learning individual-level models from heterogeneous datasets. PR estimates a unique, simple model for each data point by learning a latent distance metric between samples based on their covariates, outperforming traditional and state-of-the-art methods in prediction and interpretability across diverse applications.

What Problem Does This Paper Address?

Modern datasets are increasingly complex, often comprising data from various overlapping subgroups (e.g., patients with different disease subtypes, customers with diverse preferences). Traditional machine learning models, which aim to learn a single, global model, struggle to capture the "localized effects" – the unique patterns and relationships that are highly predictive at the individual or subgroup level. This limitation hinders the development of truly personalized policies, treatments, and predictions. Existing methods for handling heterogeneity either fail to estimate truly individual-level effects or require prior knowledge of sample relationships (like a pre-defined network), which is often unavailable.

Methods

The core of the proposed method, Personalized Regression (PR), is to estimate a unique model for each sample, denoted as $Y^{(i)} = f(X^{(i)}; \theta^{(i)})$, where $\theta^{(i)}$ is the sample-specific parameter vector. To manage the resulting over-parameterization, PR employs two key strategies:

  1. Low-Rank Representation: The matrix of all sample-specific parameters ($\Omega = [\theta^{(1)} | \dots | \theta^{(n)}]$) is constrained to have a low-rank structure. This is achieved by assuming $\theta^{(i)}$ can be represented as a linear combination of a shared dictionary $Q$ with sample-specific loadings $Z^{(i)}$, i.e., $\theta^{(i)} = Q^T Z^{(i)}$. This reduces the number of parameters and allows for efficient computation and generalization.
  2. Distance-Matching Regularization (DMR): Instead of assuming a known similarity network between samples, PR learns a latent distance metric. It uses unmodeled covariates $U^{(i)}$ to infer sample similarity. The DMR scheme penalizes the difference between the distance of samples in the covariate space and their distance in the learned latent parameter space (specifically, the $Z$ space). This encourages samples with similar covariates to have similar model parameters.

The overall objective function combines a standard loss function (e.g., least-squares, logistic loss) for each sample with regularization terms for the low-rank structure and the distance-matching objective. Optimization is performed using subgradient descent.

Datasets

The paper evaluates PR on several datasets:

  • Simulated Data: Used to assess the ability of PR to recover true effect sizes, particularly in scenarios with discontinuous parameter distributions.
  • Financial Data: A dataset of security prices over 30 years, incorporating trading history and global news headlines as predictors, and date/security characteristics as covariates.
  • Biomedical Data: Desorption electrospray ionization mass spectrometry imaging (DESI-MSI) data for basal cell carcinoma (BCC) patients, used for malignant vs. benign lesion classification.
  • Electoral Data: Data from the 2012 US presidential election, using candidate positions as predictors and county-level vote proportions as outcomes, with US Census demographic data as covariates.

Key Insights and Ideas

  • Learning Similarity from Data: A crucial innovation is learning sample relationships (similarity) from observed covariates ($U$) rather than assuming a pre-defined network. This makes the approach more broadly applicable.
  • Interpretable Embeddings: The learned sample-specific parameters ($\theta^{(i)}$) or their low-rank representations ($Z^{(i)}$) can serve as "embeddings" for each sample. These embeddings capture complex relationships and can be visualized to reveal meaningful clusters and structures (e.g., financial models clustered by time, cancer patients by subtype, counties by voting behavior and demographics).
  • Bridging Prediction and Interpretation: PR aims to achieve both high predictive accuracy and fine-grained interpretability. The sample-specific models offer insights into subgroup dynamics that are lost in global models.
  • "Overfitted Atoms" for Better Prediction: The paper draws parallels to the idea that optimal ensembles can consist of overfitted individual models, suggesting that personalized, potentially "overfitted" models can collectively lead to superior predictive performance.

Key Results

  • Simulation Study: PR significantly outperforms other methods in recovering true regression parameters, while maintaining competitive predictive error compared to deep neural networks (DNNs).
  • Financial Prediction: PR shows substantial improvements in predicting price movements, with learned models clustering by industry, region, and time, revealing how market dynamics evolve.
  • Cancer Analysis: PR achieves the best predictive performance for tumor status classification, and the derived patient-specific explanations reveal distinct subtypes, suggesting a more nuanced understanding of cancer heterogeneity than mixture models.
  • Presidential Election Analysis: PR significantly reduces out-of-sample predictive error. The learned county embeddings interpolate between demographic and voting information, providing insights into why counties with similar voting patterns might have different underlying reasons for their choices.
  • Scalability: The method is shown to be scalable, capable of fitting models to datasets with over 10,000 samples and 1000s of predictors within an hour on standard hardware.

Limitations and Future Work

  • Hyperparameter Tuning: The paper mentions that hyperparameter selection (e.g., for the regularization strengths $\lambda, \gamma, \nu$ and the latent dimensionality $q$) is discussed in the supplement, implying it's a consideration for practical application.
  • Covariate Selection for Distance: The current DMR uses unmodeled covariates $U$. The authors suggest future work could explore how to augment sample distance estimation by including distances between predictors themselves, potentially using $X$ as well.
  • Extending to Structured Models: The authors propose extending the distance-matching idea to more structured models, such as personalized probabilistic graphical models, and augmenting standard mixture models.
  • Interpreting Embeddings: While the embeddings are shown to be informative, further work could focus on developing more systematic methods for interpreting these sample-specific model representations.

Summary updated

Abstract

Modern applications of machine learning (ML) deal with increasingly heterogeneous datasets comprised of data collected from overlapping latent subpopulations. As a result, traditional models trained over large datasets may fail to recognize highly predictive localized effects in favour of weakly predictive global patterns. This is a problem because localized effects are critical to developing individualized policies and treatment plans in applications ranging from precision medicine to advertising. To address this challenge, we propose to estimate sample-specific models that tailor inference and prediction at the individual level. In contrast to classical ML models that estimate a single, complex model (or only a few complex models), our approach produces a model personalized to each sample. These sample-specific models can be studied to understand subgroup dynamics that go beyond coarse-grained class labels. Crucially, our approach does not assume that relationships between samples (e.g. a similarity network) are known a priori. Instead, we use unmodeled covariates to learn a latent distance metric over the samples. We apply this approach to financial, biomedical, and electoral data as well as simulated data and show that sample-specific models provide fine-grained interpretations of complicated phenomena without sacrificing predictive accuracy compared to state-of-the-art models such as deep neural networks.
BibTeX
@inproceedings{lengerich2019learning,
  author = {Benjamin J. Lengerich and Bryon Aragam and Eric P. Xing},
  title = {{Learning Sample-Specific Models with Low-Rank Personalized Regression}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
}
Figure