See the Lives of Stars in a Handful of Light

Shuhong Liu, Gengjia Chang, Quanfeng Xu, Ziteng Cui, Qianyu Zhou, Xiaoyu Dong, Lin Gu, Tatsuya HaradaarXiv 2026 · arXiv

TL;DR

Cosmos4D is a novel physics-informed framework that reconstructs the 800-million-year evolution of a galaxy from multi-view images. By replacing arbitrary motion fields with gravitational orbit integration and "luminosity-stratified" dynamics, it recovers star-level details and physically accurate motion where standard dynamic scene methods fail.

What Problem Does This Paper Address?

Galaxies are the ultimate "slow cinema"—their evolution spans hundreds of millions of years, far beyond a human lifetime or the history of modern telescopes. While cosmological simulations can generate these sequences, reconstructing a galaxy's 4D (3D space + time) trajectory from images alone is a massive challenge for computer vision.

Existing dynamic scene reconstruction methods (like 4D Gaussian Splatting or NeRF variants) struggle here for three reasons:

  1. Scale and Sparsity: Galaxies consist of millions of stars that appear as tiny point sources only a few pixels wide.
  2. Optical Transparency: Unlike solid objects, galaxies are "optically thin." Stars don't occlude each other; their light simply adds up, leaving no surfaces or depth orders to anchor traditional correspondence algorithms.
  3. Physical Lawlessness: Standard methods use "free deformations" to move pixels. In a galaxy, this results in stars drifting in ways that violate gravity, creating unphysical "streaks" or blurred glows instead of distinct spiral arms.

Methods

The authors introduce Cosmos4D, which treats the galaxy as a collection of self-luminous Gaussians governed by the laws of physics. Key innovations include:

  • Gravitational 4D Gaussian Splatting: Instead of learning a random displacement for each star, the model integrates stellar positions through a differentiable orbit integrator. The motion is driven by a fixed gravitational potential (representing the galaxy's bulge, disk, and dark matter halo) plus a learned, low-dimensional residual force field shared by all stars.
  • Luminosity-Stratified Dynamics: The model treats stars differently based on brightness. Bright stars are modeled with individual orbits. Faint stars, which represent unresolved populations, are carried by a shared velocity field estimated from multi-view optical flow.
  • Optically Thin Rendering: The framework uses additive splatting (summing light) rather than alpha-compositing, reflecting the true nature of galactic light.
  • Multi-Scale Supervision: To solve the "vanishing gradient" problem—where a tiny star provides no signal once it drifts a few pixels away from its target—the authors supervise the reconstruction across an image pyramid. This ensures that even "lost" stars are pulled back toward their correct positions.

Datasets

The authors curated a new 4D dataset consisting of eight evolving galaxies sourced from four major cosmological simulations: TNG50, TNG100, TNG300, and Illustris.

  • Each sequence spans 800 million years (0.8 Gyr) across 96 frames.
  • Galaxies are rendered from 14 virtual camera views.
  • This dataset provides a rigorous benchmark for both photometric rendering quality and physical accuracy (like rotation curves and angular momentum).

Key Insights and Ideas

  • Gravity as a Regularizer: The most striking insight is that physical constraints actually make the reconstruction easier and more accurate. By restricting motion to a family of gravitational orbits, the search space for the optimization is drastically reduced, preventing the "streaky" artifacts seen in unconstrained models like 4DGS.
  • The Point Source Challenge: The paper highlights a unique optimization hurdle: point sources have tiny "basins of convergence." The multi-scale approach is not just an optimization trick here; it’s a necessity for tracking individual stars over cosmic timescales.
  • Stratification is Key: By separating bright individual stars from the "diffuse glow" of faint populations, the model balances computational efficiency with high-frequency detail.

Key Results

Cosmos4D significantly outperforms state-of-the-art dynamic Gaussian baselines (such as 4DGS and 4C4D):

  • Photometric Quality: It achieved a masked LPIPS (perceptual error) score that is 39% lower than the best baseline, preserving star-level textures and sharp spiral arms that others blurred.
  • Physical Accuracy: The model reduced the error in recovered rotation curves by 43% compared to the best baseline.
  • Conservation Laws: It conserved angular momentum significantly better, with a drift error 71% lower than the best-performing baseline, proving that the recovered motion isn't just visually plausible—it's physically grounded.

Limitations and Future Work

  • Multi-View Requirement: Currently, the model requires multiple calibrated views over time. Real-world telescopes only provide one view at one instant. The authors suggest that future work could involve generative models to synthesize the missing views/epochs, which Cosmos4D could then constrain physically.
  • Interstellar Dust: The current model assumes galaxies are perfectly transparent. In reality, dust lanes absorb and scatter light, which would require a more complex, wavelength-dependent emission and absorption model.
  • Sensor Noise: The experiments used clean, simulated images. Real astronomical data is plagued by sensor noise and atmospheric/instrumental blurring (PSF), which would need to be integrated into the rendering loss.

Reference: Liu, S., et al. "See the Lives of Stars in a Handful of Light." arXiv:2611.20089v1 [cs.CV].

Summary updated

Abstract

Galaxies evolve over hundreds of millions of years, and no telescope has ever watched one move. Cosmological simulations make this evolution visible as a sequence of multi-view images, inviting the question of whether a galaxy can be recovered in four dimensions from its images. This setting defeats existing dynamic scene methods, since millions of stars span only a few pixels each, faint populations blend into diffuse light, and free deformations reproduce the images without obeying gravity. We present Cosmos4D, which rests on two components. Gravitational 4D Gaussian Splatting represents each star as a self-luminous Gaussian whose motion is integrated through a gravitational potential and a low-dimensional force field shared by all stars. Luminosity-stratified dynamics give bright stars individual orbits supervised across image scales and carry faint populations with a velocity field estimated from optical flow. We further build a 4D dataset of eight galaxies from four cosmological simulations, on which Cosmos4D outperforms dynamic Gaussian baselines in both photometric quality and physical accuracy. With these advancements, we glimpse eight hundred million years of a galaxy's life in ninety-six frames.
BibTeX
@misc{liu2026see,
  author = {Shuhong Liu and Gengjia Chang and Quanfeng Xu and Ziteng Cui and Qianyu Zhou and Xiaoyu Dong and Lin Gu and Tatsuya Harada},
  title = {{See the Lives of Stars in a Handful of Light}},
  year = {2026},
  eprint = {2611.20089},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20089},
}
Figure