Large, fast and accurate HI intensity maps with latent overlap diffusion

Sumeet Mishra, Roberto Trotta, Matteo Viel

TL;DR

Researchers have developed a two-step machine learning pipeline, HALOgen and LODI, that generates highly accurate 21 cm neutral hydrogen (HI) intensity maps from computationally "cheap" dark matter-only simulations. By using a novel "latent overlap" technique, the system can stitch together small-scale diffusion model outputs into massive, seamless cosmic volumes in just minutes, achieving 90% accuracy even deep into the non-linear regime.

What Problem Does This Paper Address?

To understand the evolution of our universe, cosmologists look for neutral hydrogen (HI). This gas emits a specific radio signal—the 21 cm line—which acts as a tracer for the underlying distribution of dark matter and the "cosmic web."

However, predicting what these signals should look like requires massive hydrodynamic simulations that model complex baryonic physics (like star formation and gas cooling). These simulations are prohibitively expensive, often taking millions of CPU hours to produce even modest volumes. Dark matter-only (N-body) simulations are much faster but don't include the gas physics needed to predict the 21 cm signal. The challenge is: how do we "paint" the 21 cm signal onto cheap dark matter simulations accurately and at scale?

Methods

The authors propose a modular, two-stage generative pipeline:

  1. HALOgen (Halo Assignment and Generation): An attention-based 3D ResUNet. It takes a dark matter density map as input and predicts the density of dark matter halos across four different mass bins. This step bridges the gap between raw matter and the "buckets" (halos) where hydrogen lives.
  2. LODI (Latent Overlap Diffusion for Intensity mapping): A conditional variational diffusion model. It takes the halo maps from HALOgen and "inpaints" the 21 cm brightness temperature. Diffusion models are excellent at capturing the stochastic, non-linear nature of small-scale structures.

The "Latent Overlap" Innovation: Standard ML models are limited by GPU memory and can only process small sub-volumes. Simply tiling these sub-volumes creates "seams" or discontinuities at the boundaries. The authors introduced Latent Overlap, where adjacent sub-volumes are generated such that they share a small overlapping region. During the diffusion (denoising) process, the model uses the already-generated "latent" state of a neighbor to guide the generation of the current block, ensuring a perfectly seamless transition across the entire cosmic volume.

Datasets

The models were trained and validated using the CAMELS (Cosmology and Astrophysics with MachinE Learning Simulations) dataset, specifically the IllustrisTNG suite.

  • Training: 5 simulations with different initial seeds but the same cosmology.
  • Testing: Held-out simulations with unseen initial seeds to ensure the model wasn't just "memorizing" the maps.
  • Resolution: The volumes were $25^3 (Mpc/h)^3$ with $256^3$ voxels, providing a resolution of $\approx 0.1 \, h^{-1} \text{Mpc}$.

Key Insights and Ideas

  • Separation of Concerns: By splitting the task into "Matter $\to$ Halos" and "Halos $\to$ HI," the model respects the physical hierarchy of the universe. HALOgen handles the large-scale structure, while LODI handles the complex, small-scale gas physics.
  • Diffusion for Small Scales: Unlike standard U-Nets which can produce "blurry" results, the diffusion model (LODI) excels at generating the high-frequency, non-linear fluctuations essential for accurate power spectra at small scales ($k > 1$).
  • Scalability via Inpainting: The Latent Overlap method effectively treats the generation of the universe as a massive, multi-step inpainting problem. This allows the model to scale to arbitrarily large volumes (like those expected from the Square Kilometre Array) without needing a single massive GPU.

Key Results

  • Accuracy: The pipeline predicts the 21 cm power spectrum to within 10% accuracy for wavenumbers $k \le 10 \, h \, \text{Mpc}^{-1}$. This is significant because it covers the "non-linear regime" where traditional analytical methods usually fail.
  • Speed: Generating a full $256^3$ voxel temperature map takes approximately 2 minutes on a single NVIDIA A100 GPU. This represents a massive speedup compared to traditional hydrodynamic simulations.
  • Visual Fidelity: The Latent Overlap method successfully eliminated the "grid artifacts" typically seen in tiled generative models, producing a continuous cosmic web.

Limitations and Future Work

  • Fixed Cosmology: This study was a proof-of-concept using a fixed set of cosmological parameters ($\Omega_m, \sigma_8$, etc.). The authors acknowledge that the next step is training the model on the broader CAMELS set to allow it to generalize across different universes.
  • Redshift: The current results are focused on $z=0$. Future iterations will need to produce "lightcones"—maps that account for the change in the universe's state as the signal travels across billions of years to reach our telescopes.
  • Observational Realism: The current model produces "perfect" maps. Future work will need to incorporate instrumental effects like beam smoothing, radio interference, and foreground contamination to be directly useful for telescope data pipelines.

Summary updated

Abstract

The distribution of 21 cm emission from neutral hydrogen is a powerful cosmological and astrophysical probe, as it traces the underlying dark matter and cold gas distributions throughout cosmic times. However, the prediction of observable signals is hindered by the large computational costs of the required hydrodynamic simulations. We introduce a novel machine learning pipeline that, once trained on a hydrodynamical simulation, is able to generate both halo mass density maps and the three-dimensional 21 cm brightness temperature signal, starting from a dark matter-only simulation. We use an attention-based ResUNet (HALO) to predict dark matter halo maps, which are then processed through a trained conditional variational diffusion model (LODI) to produce 21 cm brightness temperature maps. LODI is trained on smaller sub-volumes that are then seamlessly combined in 512-times larger volume using a new method, called `latent overlap'. We demonstrate that, once trained on 25^3 (Mpc/h)^3 volume simulations, we are able to predict the 21 cm power spectrum on an unseen dark matter map (with the same cosmology) to within 10% for wavenumbers k <= 10 h Mpc^-1, deep inside the non-linear regime, with a computational effort of the order of two minutes. While demonstrated on this specific volume, our approach is designed to be scalable to arbitrarily large simulations.
BibTeX
@inproceedings{mishra2025large,
  author = {Sumeet Mishra and Roberto Trotta and Matteo Viel},
  title = {{Large, fast and accurate HI intensity maps with latent overlap diffusion}},
  booktitle = {Monthly},
  year = {2025},
  doi = {10.1093/mnras/staf2071},
}
Figure