Converting standard-dynamic-range (SDR) video to high-dynamic-range (HDR) is a classic "ill-posed" problem. You’re trying to recover details that were lost to clipped highlights or crushed, noisy shadows. While modern latent diffusion models (LDMs) are excellent at "hallucinating" missing details, a new paper from researchers at STUDIO FREEWILLUSION and Seoul National University reveals a hidden bottleneck: the Variational Autoencoder (VAE).
TL;DR
The authors identify "dark-side collapse," where standard HDR encodings cause color information to vanish in low-light regions when passed through a frozen VAE. They introduce EquivFlow, an invertible, exposure-equivariant encoding that preserves chromatic detail in shadows, leading to significantly better color accuracy in SDR-to-HDR reconstruction.
What Problem Does This Paper Address?
Most state-of-the-art generative models for video don't work directly on pixels; they work in a compressed "latent space" created by a VAE. To use these models for HDR, you have to map high-intensity HDR signals into the VAE’s limited operating range.
The standard approach is to use log-like transforms (like LogC3). However, the authors discovered that these transforms cause latent chromatic structure collapse. In plain English: as an image gets darker, the different colors in the latent space get squashed into a single point. By the time the diffusion model sees the data, the color information in the shadows is already gone.
Methods
The researchers propose EquivFlow, a compact, 13-parameter encoding designed to be "VAE-aware."
- Exposure Equivariance: In linear HDR, changing exposure is a multiplicative operation. EquivFlow is designed so that this multiplicative change in pixel space becomes a simple additive shift (translation) along a specific "exposure axis" in the latent space. This ensures that the "chroma plane" (the color information) remains scale-independent and doesn't contract as things get darker.
- VAE-Aware Optimization: Instead of using a fixed formula from color science, the parameters of EquivFlow are fitted specifically to a frozen VAE. They use a "round-trip" loss to ensure that an image encoded and then decoded by the VAE retains as much detail as possible.
- Two-Step Pipeline:
- Step 1: Fit the EquivFlow encoding to the frozen VAE.
- Step 2: Train a lightweight LoRA adapter for a frozen Diffusion Transformer (DiT) backbone to perform the actual SDR-to-HDR reconstruction in this optimized latent space.
Datasets
- Training: A custom corpus of 5,780 video crops derived from Tears of Steel (Blender Foundation) and HDR panoramas from PolyHaven.
- Evaluation:
- ARRI: Official ALEXA Mini sample reel (high-end cinema footage).
- HDRTV4K: A dataset of mixed video and static content.
- UPIQ: A dataset of calibrated static images. The authors used these to test how well the model generalizes to "unseen" tone-mapping curves (like ACES).
Key Insights and Ideas
- The VAE is the Bottleneck: Even if your diffusion model is perfect, it can't recover color if the VAE latent space has already collapsed it.
- Symmetry is Overrated: The paper finds that the loss of chromatic structure is asymmetric; it's much worse in the dark (underexposure) than in the highlights (overexposure) for standard VAEs.
- Portability: Surprisingly, an EquivFlow encoding fitted to one VAE (like LTX-Video) shows "training-free portability" to other VAEs (like Stable Diffusion 2.1 or Wan 2.1). The exposure-linked latent displacement remains nearly linear across different architectures.
Key Results
EquivFlow consistently outperformed existing baselines like LumiVid and HDRTVNet++ in chromatic fidelity.
- Quantitative: In the UPIQ dataset, EquivFlow achieved a $\Delta C_{T}C_{P}$ (a measure of perceptual color error) of 3.02, compared to 3.93 for LogC3 and 6.18 for LumiVid. Lower is better, and these are significant gains in color accuracy.
- Qualitative: In side-by-side comparisons, EquivFlow reconstructions maintain the correct hue and saturation in deep shadows where other models produce grayed-out or incorrectly colored pixels.
Limitations and Future Work
- Tone-Mapping Dependence: While EquivFlow improves color, the overall reconstruction quality (PSNR) still depends heavily on the specific SDR tone-mapping curve used during training.
- Fixed VAEs: The method is designed for frozen VAEs. While this is practical for using existing large-scale models, future work could explore joint training of the encoding and the VAE itself.
- Scope: The current study focuses on video reconstruction; the authors suggest that exploring these "dark matters" in broader VAE families and different generative tasks is a promising next step.
Summary updated