Learning Sample-Specific Models with Low-Rank Personalized Regression

Benjamin J. Lengerich, Bryon Aragam, Eric P. Xing

In the era of Big Data, we often assume that more data automatically leads to better models. However, as datasets grow, they become increasingly heterogeneous—composed of diverse subpopulations that react differently to the same stimuli. A "global" model trained on everyone might end up being a mediocre model for everyone.

A 2019 paper by Benjamin Lengerich, Bryon Aragam, and Eric P. Xing, titled "Learning Sample-Specific Models with Low-Rank Personalized Regression," introduces a framework to solve this by giving every single data point its own unique model.

The Problem with Global Models

Traditional machine learning usually seeks a single function $f(X; \theta)$ that fits the entire population. When data is heterogeneous—like a medical dataset containing patients with different genetic backgrounds—this global approach fails. Localized effects, which are crucial for precision medicine or targeted advertising, get "washed out" by the majority signal.

Existing solutions have major drawbacks:

  1. Mixture Models: Only estimate a few sub-models, failing to capture individual-level nuances.
  2. Varying Coefficient Models: Require the relationships between samples to be smooth and well-defined.
  3. Network Lasso: Requires you to already know the "similarity network" between samples (e.g., a social graph) before you start.

The challenge is: How do we estimate $n$ different models for $n$ samples without overfitting or requiring a pre-defined map of how those samples relate?

The Solution: Personalized Regression (PR)

The authors propose Personalized Regression (PR), a method that estimates a unique, simple model for every individual sample in a dataset. By using low-rank constraints and learning a latent distance metric between samples, PR captures fine-grained local patterns that global models (and even deep neural networks) often miss, significantly improving both interpretability and predictive accuracy.

The core idea is to let the parameters $\theta$ vary for every sample $i$, such that the model becomes $Y^{(i)} = f(X^{(i)}; \theta^{(i)})$. To make this mathematically feasible and avoid overfitting, the authors introduce key constraints:

  • Low-Rank Representation: They assume the matrix of all personalized parameters is low-rank. Every individual's model is a unique combination of a small, shared "dictionary" of base patterns, allowing models to "borrow power" from one another.
  • Distance-Matching Regularization (DMR): Instead of requiring a known network, the authors use auxiliary covariates ($U$)—data you have but don't want to use as direct predictors—to learn a latent distance metric. If two samples are "close" in this latent space, their regression parameters $\theta$ should also be similar.
  • Anchored Optimization: Models are initialized around a standard "population" model (like a Lasso regression). The "center of mass" of these personalized models stays close to the population model, ensuring the system doesn't diverge into chaos.

Key Results and Datasets

The authors evaluated PR across four distinct environments, proving its efficacy over traditional methods.

Comparison of modeling approaches on simulated data.
  • Simulated Data: As seen in the figure above, PR successfully recovers true, discontinuous effect sizes. In contrast, Mixture Models, Varying-Coefficient Models, and even Deep Neural Networks (DNNs) struggle to capture these localized, non-linear patterns.
  • Financial Data: Using 30 years of stock/ETF prices paired with global news headlines, PR was a standout performer. In stock price prediction, it achieved an $R^2$ of 0.86, whereas the standard population model and DNNs hovered near zero or were negative. This suggests financial markets are so non-stationary that only a personalized, time-aware model can track them.
  • Cancer Analysis: On a mass spectrometry imaging dataset of 17,053 samples from skin cancer patients, PR achieved the highest AUROC (0.923) and accuracy (0.975) in distinguishing malignant from benign tissue, outperforming mixture models and DNNs.
  • Electoral Data: Using county-level results from the 2008 and 2012 U.S. Presidential elections, PR successfully "interpolated" between demographic data and voting outcomes, providing a much clearer picture of regional political shifts.

Key Insights

  • Interpretability via Embeddings: By performing t-SNE on the learned parameters ($\theta^{(i)}$), the authors created "model embeddings." In the election study, this revealed that counties with similar demographics might vote the same way for entirely different reasons—a nuance invisible to standard models.
  • The "Spring" Analogy: The distance-matching regularizer acts like a collection of springs. They pull similar models together but don't move the overall "center of mass," allowing for local flexibility without losing global stability.
  • Accuracy vs. Inference: Deep Neural Networks are great at prediction but often fail to recover the true underlying mechanics of the data. PR achieves DNN-level accuracy while remaining as interpretable as a simple linear regression.

Limitations and Future Work

While powerful, PR has limitations. Calculating pairwise distances for very large datasets ($N$) is computationally intensive. Furthermore, the success of the personalization depends heavily on having good auxiliary covariates ($U$) to define the latent space; if $U$ is noisy or irrelevant, the personalization might not be meaningful.

Future directions include extending this framework to more complex structures, such as Personalized Probabilistic Graphical Models, to understand how relationships between variables change across individuals.

Summary updated

Abstract

Modern applications of machine learning (ML) deal with increasingly heterogeneous datasets comprised of data collected from overlapping latent subpopulations. As a result, traditional models trained over large datasets may fail to recognize highly predictive localized effects in favour of weakly predictive global patterns. This is a problem because localized effects are critical to developing individualized policies and treatment plans in applications ranging from precision medicine to advertising. To address this challenge, we propose to estimate sample-specific models that tailor inference and prediction at the individual level. In contrast to classical ML models that estimate a single, complex model (or only a few complex models), our approach produces a model personalized to each sample. These sample-specific models can be studied to understand subgroup dynamics that go beyond coarse-grained class labels. Crucially, our approach does not assume that relationships between samples (e.g. a similarity network) are known a priori. Instead, we use unmodeled covariates to learn a latent distance metric over the samples. We apply this approach to financial, biomedical, and electoral data as well as simulated data and show that sample-specific models provide fine-grained interpretations of complicated phenomena without sacrificing predictive accuracy compared to state-of-the-art models such as deep neural networks.
BibTeX
@inproceedings{lengerich2019learning,
  author = {Benjamin J. Lengerich and Bryon Aragam and Eric P. Xing},
  title = {{Learning Sample-Specific Models with Low-Rank Personalized Regression}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
}
Figure