StratLearn-z: Improved photo-$z$ estimation from spectroscopic data subject to selection effects

Chiara Moretti, Maximilian Autenrieth, R Serra, Roberto Trotta, David A. van Dyk, Andrei Mesinger

TL;DR

StratLearn-z is a new machine learning framework designed to solve the "selection bias" problem in estimating galaxy distances (redshifts). By using a statistical technique called propensity score stratification, it creates a more balanced training process that outperforms standard methods—reducing bias by an order of magnitude and cutting error rates in half when training data doesn't match the target survey.

What Problem Does This Paper Address?

To map the history of our Universe, astronomers need to know the distance to billions of galaxies. The most accurate way to do this is through spectroscopy, but it is incredibly time-consuming. Instead, modern surveys use "photometric redshifts" (photo-$z$), which estimate distance based on a galaxy's color and brightness across a few broad filters.

The challenge is that machine learning models for photo-$z$ need training data. Our best training data comes from spectroscopic surveys, which are heavily biased toward bright, nearby galaxies. However, the "target" surveys we want to analyze (like LSST or Euclid) are full of faint, distant galaxies. This mismatch is known as covariate shift. When a model trained on "easy" bright galaxies tries to predict "hard" faint ones, it often fails spectacularly, leading to systematic biases that can ruin cosmological measurements.

Methods

The authors introduce StratLearn-z, an application of the "StratLearn" method specifically for redshift estimation. The core innovation is a two-step process:

  1. Propensity Score Stratification: The algorithm first learns to predict how likely a galaxy is to be in the training (source) set versus the target set based on its observed features (magnitudes and colors). This probability is the "propensity score." The data is then split into five "strata" (subgroups) based on these scores. Within each stratum, the training and target galaxies look much more alike, effectively "balancing" the dataset.
  2. Ensemble Estimation: Inside each stratum, the model fits two different types of Conditional Density Estimators (CDEs): a kernel-based nearest neighbor model (ker-NN) and an eigenfunction-based Series estimator. These are combined via a weighted average to produce a final probability distribution for the redshift.

By training separate models for different "types" of galaxies (defined by their propensity to be in the training set), the system avoids the pitfalls of global covariate shift.

Datasets

The researchers used a simulated dataset from the Buzzard Flock suite, specifically the LSST-DESC DC1 mock catalog. This contains 100,000 galaxies with realistic colors and redshifts.

To test the model's robustness, they manually introduced four levels of covariate shift (None, Weak, Mild, and Strong) using a technique called rejection sampling on the $r$-band magnitude. This mimics real-world scenarios where faint galaxies are systematically missing from spectroscopic training sets.

Key Insights and Ideas

  • Stratification as a "Fix": The "aha moment" here is that while the total training set is unrepresentative, specific slices of it are representative of corresponding slices of the target set. Stratifying by propensity score is a statistically principled way to find those slices.
  • Conservative is Better than Overconfident: The model's probability distributions tend to be "fat" (conservative in variance). While this means the individual predictions are less precise, it significantly reduces "catastrophic errors"—cases where the model is confidently wrong.
  • Robustness to Bias: Unlike standard Gaussian Process methods (like the benchmarked GPz), StratLearn-z's performance metrics barely degrade as the covariate shift gets stronger. It "knows" how to handle the faint objects that usually confuse other models.

Key Results

The authors benchmarked StratLearn-z against GPz, a popular Gaussian Process-based photo-$z$ code. In the "Strong Covariate Shift" scenario:

  • Bias: StratLearn-z improved the bias by one order of magnitude compared to GPz.
  • Accuracy: It achieved a factor of 2 improvement in Root Mean Square Error (RMSE).
  • Catastrophic Errors: The fraction of objects with massive prediction errors was significantly lower.
  • High Redshift Performance: Using the Continuous Rank Probability Score (CRPS), StratLearn-z showed marked improvements at high redshifts ($z > 1.5$), where training data is scarcest.

Limitations and Future Work

The authors acknowledge a few key areas for improvement:

  • The "Redshift Desert": There are certain redshift ranges (like $1.4 \le z \le 2.5$) where even spectroscopic surveys struggle to get a signal. If the training data is missing entirely for a specific range, no amount of stratification can recover that information.
  • Assumption of Shared Physics: The method assumes that the relationship between color and redshift is the same for both bright and faint galaxies. If faint galaxies have fundamentally different physical properties (e.g., different metallicities) that aren't captured by the covariates, the model may still struggle.
  • Model Ensemble: The current implementation uses a specific pair of estimators. The authors suggest that adding other models (like GPz itself) into the StratLearn framework could potentially "flatten" the probability distributions and improve precision.

The Julia implementation of the method is publicly available on GitHub.

Summary updated

Abstract

A precise measurement of photometric redshifts (photo-z) is key for the success of modern photometric galaxy surveys. Machine learning (ML) methods show great promise in this context, but suffer from covariate shift (CS) in training sets due to selection bias where interesting sources are underrepresented, and the corresponding ML models show poor generalisation properties. We present an application of the StratLearn method to the estimation of photo-z, validating against simulations where we enforce the presence of CS to different degrees. StratLearn is a statistically principled approach that relies on splitting the source and target datasets into strata based on estimated propensity scores (i.e. the probability for an object to be in the source set given its observed covariates). After stratification, two conditional density estimators are fit separately to each stratum, then combined via a weighted average. We benchmark our results against the GPz algorithm, quantifying the performance of the two codes with a set of metrics. Our results show that the StratLearn-z metrics are only marginally affected by the presence of CS, while GPz shows a significant degradation of performance in the photo-z prediction for fainter objects. For the strongest CS scenario, StratLearn-z yields a reduced fraction of catastrophic errors, a factor of 2 improvement for the RMSE and one order of magnitude improvement on the bias. We also assess the quality of the conditional redshift estimates with the probability integral transform (PIT). The PIT distribution obtained from StratLearn-z features fat fewer outliers and is symmetric, i.e. the predictions appear to be centered around the true redshift value, despite showing a conservative estimation of the spread of the conditional redshift distributions. Our julia implementation of the method is available at https://github.com/chiaramoretti/StratLearn-z.
BibTeX
@article{moretti2025stratlearn,
  author = {Chiara Moretti and Maximilian Autenrieth and R Serra and Roberto Trotta and David A. van Dyk and Andrei Mesinger},
  title = {{StratLearn-z: Improved photo-$z$ estimation from spectroscopic data subject to selection effects}},
  journal = {The Open Journal of Astrophysics},
  year = {2025},
  doi = {10.33232/001c.137525},
}

Loading related content...

Figure