Abstract
Digitally acquired high dynamic range (HDR) video baseband signal can take 10 to 12 bits per color channel. It is economically important to be able to reuse the legacy 8 or 10-bit video codecs to efficiently compress the HDR video. Linear or nonlinear mapping on the intensity can be applied to the baseband signal to reduce the dynamic range before the signal is sent to the codec, and we refer to this range reduction step as a baseband quantization. We show analytically and verify using test sequences that the use of the baseband quantizer lowers the coding efficiency. Experiments show that as the baseband quantizer is strengthened by 1.6 bits, the drop of PSNR at a high bitrate is up to 1.60dB. Our result suggests that in order to achieve high coding efficiency, information reduction of videos in terms of quantization error should be introduced in the video codec instead of on the baseband signal.
Introduction
Modern High Dynamic Range (HDR) video acquisition often captures 10 to 12 bits per color channel to preserve a vast range of luminance and color detail. However, existing video infrastructure, including legacy codecs like H.264/AVC, is frequently optimized for 8 or 10-bit signals. To bridge this gap, a common practice is "baseband quantization" or "reshaping"—mapping the high-precision signal down to a lower bit depth before it enters the encoder. While this ensures compatibility, it introduces a critical trade-off in coding efficiency. This paper provides a formal analytical proof and experimental quantification of the quality loss associated with this pre-processing step.
Modeling the Coding Pipeline
To analyze the impact of baseband quantization, the authors model the video codec as a system with two sequential quantization steps: $Q_1$ (the baseband quantizer) and $Q_2$ (the codec's internal quantizer).
The study simplifies the complex predictive coding loop into a mathematically tractable model. Key components include:
- Residue Modeling: Proving (Lemma 1) that the error in predictive coding can be approximated by analyzing non-predictive "residues."
- Geometric Analysis: Deriving Mean Squared Error (MSE) formulas based on N-dimensional hypercubes (Lemma 2) to calculate how reconstruction centroids shift when different quantization grids are overlaid.
The theory was validated using both synthetic Gaussian AR(1) vectors and standard HDR video sequences (BALLOONFESTIVAL, MARKET, and TYPEWRITER) processed in the BT.2020 color space.
Key Insights
The analytical model reveals several fundamental limitations of the baseband quantization architecture:
- Irreducible Error Floor: The initial baseband quantizer ($Q_1$) introduces a permanent error. Even at extremely high bitrates where the codec's quantizer ($Q_2$) is very fine, the final quality is capped by the precision lost at the baseband.
- Codec Dominance: Bitrate is primarily controlled by $Q_2$. Coarsening $Q_1$ does not significantly reduce the bitrate but does lower the Signal-to-Noise Ratio (SNR).
- The 3 dB Rule: In the worst-case scenario where $Q_1$ and $Q_2$ have equal step sizes, the system suffers a maximum SNR drop of 3 dB.
- Content Independence: The efficiency loss is largely independent of block size or spatial correlation, indicating a fundamental architectural penalty.
Experimental Results
The researchers compared "precise" baseband signals (fine $Q_1$) against "coarse" ones. Experiments demonstrated that strengthening the baseband quantizer by 1.6 bits resulted in a PSNR drop of up to 1.60 dB at high bitrates.
As shown in Figure 6, the simulated SNR drops for various quantization ratios ($q_1/q_2$) align almost perfectly with the mathematical predictions. For instance, at a 1:1 ratio, the theoretical loss of -2.98 dB matches the simulated -3.00 dB. For real-world sequences like TYPEWRITER, the quality gap between precise and coarse baseband signals remained significant across all tested bitrates.
Conclusion
The findings indicate that information reduction via quantization should ideally occur within the video codec rather than on the baseband signal. While this study focused on linear mapping, future work should explore if nonlinear mappings (like PQ or HLG curves) exhibit similar efficiency penalties. The core recommendation for HDR workflows is to maintain the highest possible precision in the baseband signal to maximize final coding efficiency.
Summary updated