TL;DR
This paper identifies a systematic "luminance error" in generative inverse tone mapping (ITM) where reconstructions are biased—leaning too bright or too dark—depending on the unknown tone operator used to create the SDR input. By reframing ITM as blind posterior inference and supervising log-luminance directly in a diffusion model's latent space with an asymmetric loss, the authors achieve state-of-the-art HDR video reconstruction that is remarkably robust across different tone-mapping styles.
What Problem Does This Paper Address?
High-dynamic-range (HDR) displays are now standard, but the vast majority of video content—from archival footage to the output of modern AI video generators—exists only in standard dynamic range (SDR). Converting SDR to HDR (Inverse Tone Mapping, or ITM) is a "blind inverse problem." The original tone-mapping operator (TMO) that compressed the scene's light into an 8-bit signal is unknown, and "clipping" (where bright areas are pushed to pure white) permanently destroys information.
Recent methods use generative diffusion models to "hallucinate" this lost information. However, the authors discovered a critical flaw: these models are systematically biased. Depending on which TMO produced the input, the reconstructed HDR might be consistently too bright or too dark. Standard metrics often hide this because they only measure the magnitude of error, not its direction (sign), or they use alignment techniques that mask the bias.
Methods
The authors propose a two-pronged strategy to fix this "luminance error":
- Latent Luminance Supervision: The researchers found a surprising "aha moment": inside the latent space of a frozen video autoencoder (LTX-2), log-luminance is represented by a specific, nearly linear direction. They fit a simple affine "probe" to read this coordinate without needing to decode the image.
- Asymmetric Loss (Expectile Action): Instead of a standard loss that targets the average (mean) of all possible HDR values, they use an asymmetric quadratic loss on the luminance coordinate. This allows them to select a specific "expectile" of the posterior distribution. This acts as a "bias dial," specifically correcting the model's tendency to lean bright or dark in clipped regions.
- Observability Gating: This correction is only applied where the signal is clipped (low "observability"). In unclipped regions, the model follows the direct supervision of the input.
- Operator-Marginalized Training: Rather than training on a single tone-mapping style, they train the model on a diverse family of five different operators (Hard-Clip, ACES, Reinhard, etc.). This teaches the model to be robust to whatever unknown operator it might encounter at test time.
Datasets
- Training: A corpus of 347 source clips (187 real-world captures from HdM-HDR-2014 and 160 synthetic renders). These were expanded into 1,735 training pairs using the five-operator TMO family.
- Evaluation Benchmarks: The authors tested on three distinct scenarios:
- Hard-Clip: Unseen content using a standard hard-clipping operator.
- Filmic: Unseen content using a "soft" filmic curve (Hable).
- Severe: An intentionally difficult, out-of-distribution operator with extreme exposure and gamma settings.
- Test Sources: 84 clips from ARRI, UPIQ (photometrically calibrated stills), and LiU HDRv datasets.
Key Insights and Ideas
- Luminance is a Latent Handle: It is non-trivial that a complex, non-linear video autoencoder preserves a nearly linear "luminance axis" in its latent space. This allows for "decode-free" supervision during training, which is computationally efficient and direct.
- Bias is a Choice: In clipped regions, there isn't one "correct" answer, but a range of plausible ones (the posterior). A model's systematic bias is essentially a bad choice from that range. By using an asymmetric loss, we can explicitly steer the model toward a more neutral "expectile" rather than a biased mean.
- Marginalization over Inference: Many researchers try to infer the tone operator at test time. This paper argues that it's more effective to marginalize over them—training the model to produce stable results regardless of the operator, as long as it falls within a broad family of styles.
Key Results
- Lowest Error: The proposed system achieved the lowest luminance error (RMSE) of any public method across all benchmarks (0.141 mean vs. 0.179–0.341 for baselines).
- Neutral Bias: While baseline methods swung wildly between being too bright (+0.09) and too dark (-0.26), the authors' method stayed remarkably neutral, with a signed bias between +0.01 and +0.04.
- Perceptual Superiority: The method took first place on all three benchmarks for perceptual quality (CVVDP JOD) and fidelity (PU21-PSNR).
- Visual Fidelity: Qualitative results (see Figure 2) show that when "stopping down" (re-exposing) the reconstructed HDR, the proposed method recovers plausible structural detail in highlights where others produce flat, constant colors or noisy artifacts.
Limitations and Future Work
- The "Severe" Boundary: The method's robustness is tied to the "span" of its training family. Performance degrades on the "Severe" benchmark, which uses settings far outside the training distribution. Expanding the operator family is a clear next step.
- Unverifiable Content: Because clipped regions are truly lost, the "correctness" of hallucinated details is subjective. The authors focus on constraining the direction of luminance, but the specific textures generated are just one plausible sample from the model's prior.
- Backbone Specificity: The luminance probe was specifically calibrated for the LTX-2 architecture. While the authors showed some transferability to other models, a new probe would likely be needed for significantly different architectures.
Summary updated