Dark Matters: VAE-Aware HDR Encoding for SDR-to-HDR Video Reconstruction

Donghoon Cha, Bomseok Park, Hansl von Kwon, Ku DoHyung, Hanwool SularXiv 2026 · arXiv

Converting standard-dynamic-range (SDR) video to high-dynamic-range (HDR) is a classic "ill-posed" problem. You’re trying to recover details that were lost to clipped highlights or crushed, noisy shadows. While modern latent diffusion models (LDMs) are excellent at "hallucinating" missing details, a new paper from researchers at STUDIO FREEWILLUSION and Seoul National University reveals a hidden bottleneck: the Variational Autoencoder (VAE).

TL;DR

The authors identify "dark-side collapse," where standard HDR encodings cause color information to vanish in low-light regions when passed through a frozen VAE. They introduce EquivFlow, an invertible, exposure-equivariant encoding that preserves chromatic detail in shadows, leading to significantly better color accuracy in SDR-to-HDR reconstruction.

What Problem Does This Paper Address?

Most state-of-the-art generative models for video don't work directly on pixels; they work in a compressed "latent space" created by a VAE. To use these models for HDR, you have to map high-intensity HDR signals into the VAE’s limited operating range.

The standard approach is to use log-like transforms (like LogC3). However, the authors discovered that these transforms cause latent chromatic structure collapse. In plain English: as an image gets darker, the different colors in the latent space get squashed into a single point. By the time the diffusion model sees the data, the color information in the shadows is already gone.

Figure 1: Motivation. Left: Frozen-VAE latent PCA for 12 chromaticities over 0 to −8 EV. LogC3 trajectories collapse to one point at −8 EV, while the proposed exposure-equivariant encoding keeps them parallel and separated. The frozen autoencoder and color samples are identical, isolating the encoding as the cause. Right: The same failure in a close-up: the EV+0 frame, the crushed region lifted to EV+16, and its CIE 1976 u′v′ pixel distribution relative to the Rec.709 gamut.

Methods

The researchers propose EquivFlow, a compact, 13-parameter encoding designed to be "VAE-aware."

  1. Exposure Equivariance: In linear HDR, changing exposure is a multiplicative operation. EquivFlow is designed so that this multiplicative change in pixel space becomes a simple additive shift (translation) along a specific "exposure axis" in the latent space. This ensures that the "chroma plane" (the color information) remains scale-independent and doesn't contract as things get darker.
  2. VAE-Aware Optimization: Instead of using a fixed formula from color science, the parameters of EquivFlow are fitted specifically to a frozen VAE. They use a "round-trip" loss to ensure that an image encoded and then decoded by the VAE retains as much detail as possible.
  3. Two-Step Pipeline:
    • Step 1: Fit the EquivFlow encoding to the frozen VAE.
  4. Step 2: Train a lightweight LoRA adapter for a frozen Diffusion Transformer (DiT) backbone to perform the actual SDR-to-HDR reconstruction in this optimized latent space.
Figure 2: Method overview. The display-referred SDR observation is mapped to a scene-linear proxy by a fixed preprocessing convention, so observation and HDR target share one scene-linear coordinate system; the invertible exposure-equivariant encoding fθ then carries that space into the pretrained VAE, and is what the generative model actually sees. The encoding is fitted first against the frozen VAE, and the adapter is trained afterwards in the resulting code space. The VAE and the diffusion backbone stay frozen throughout.

Datasets

  • Training: A custom corpus of 5,780 video crops derived from Tears of Steel (Blender Foundation) and HDR panoramas from PolyHaven.
  • Evaluation:
    • ARRI: Official ALEXA Mini sample reel (high-end cinema footage).
  • HDRTV4K: A dataset of mixed video and static content.
  • UPIQ: A dataset of calibrated static images. The authors used these to test how well the model generalizes to "unseen" tone-mapping curves (like ACES).

Key Insights and Ideas

  • The VAE is the Bottleneck: Even if your diffusion model is perfect, it can't recover color if the VAE latent space has already collapsed it.
  • Symmetry is Overrated: The paper finds that the loss of chromatic structure is asymmetric; it's much worse in the dark (underexposure) than in the highlights (overexposure) for standard VAEs.
  • Portability: Surprisingly, an EquivFlow encoding fitted to one VAE (like LTX-Video) shows "training-free portability" to other VAEs (like Stable Diffusion 2.1 or Wan 2.1). The exposure-linked latent displacement remains nearly linear across different architectures.

Key Results

EquivFlow consistently outperformed existing baselines like LumiVid and HDRTVNet++ in chromatic fidelity.

  • Quantitative: In the UPIQ dataset, EquivFlow achieved a $\Delta C_{T}C_{P}$ (a measure of perceptual color error) of 3.02, compared to 3.93 for LogC3 and 6.18 for LumiVid. Lower is better, and these are significant gains in color accuracy.
  • Qualitative: In side-by-side comparisons, EquivFlow reconstructions maintain the correct hue and saturation in deep shadows where other models produce grayed-out or incorrectly colored pixels.
Figure 4: Qualitative comparison. EquivFlow remains closer to the target's color and structure in dark regions compared to LumiVid and HDRTVNet++.

Limitations and Future Work

  • Tone-Mapping Dependence: While EquivFlow improves color, the overall reconstruction quality (PSNR) still depends heavily on the specific SDR tone-mapping curve used during training.
  • Fixed VAEs: The method is designed for frozen VAEs. While this is practical for using existing large-scale models, future work could explore joint training of the encoding and the VAE itself.
  • Scope: The current study focuses on video reconstruction; the authors suggest that exploring these "dark matters" in broader VAE families and different generative tasks is a promising next step.

Summary updated

Abstract

SDR-to-HDR reconstruction is an ill-posed problem in which clipped highlights, crushed shadows, and quantization artifacts must be recovered while preserving the original content. Latent diffusion models provide powerful generative priors for restoring missing detail, but their effectiveness is constrained by how HDR signals pass through the pretrained VAE. We find that common log-like encodings suppress low-luminance chromatic differences before generation, producing a dark-side collapse in the frozen-VAE latent. We introduce EquivFlow, a compact, invertible exposure-equivariant encoding that maps exposure changes to additive shifts while retaining a scale-independent chromatic component, without modifying the VAE. Encoding-level tests across pretrained image and video VAEs show that its exposure structure and deep-dark chroma transport persist across the tested hosts. Integrated into SDR-to-HDR video diffusion, EquivFlow delivers consistent chromatic-fidelity gains across the tested settings while retaining competitive reconstruction fidelity.
BibTeX
@misc{cha2026dark,
  author = {Donghoon Cha and Bomseok Park and Hansl von Kwon and Ku DoHyung and Hanwool Sul},
  title = {{Dark Matters: VAE-Aware HDR Encoding for SDR-to-HDR Video Reconstruction}},
  year = {2026},
  eprint = {2611.13504},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13504},
}
Figure