TL;DR
This paper introduces KPN-MFI, a novel deep learning framework for converting standard dynamic range (SDR) videos to high dynamic range (HDR). It addresses the challenge of maintaining temporal consistency in video HDR conversion by introducing a multi-frame interaction (MFI) module that leverages information from adjacent frames, significantly improving upon existing single-frame methods.
What Problem Does This Paper Address?
High dynamic range (HDR) displays offer a more realistic visual experience, but the vast majority of available video content is in standard dynamic range (SDR). Converting SDR video to HDR (inverse tone mapping or iTM) is crucial for enjoying HDR content on modern displays. While image-based iTM methods have seen significant progress, extending them to video is challenging. Directly applying image iTM models to video frames independently leads to flickering and temporal inconsistencies, as these methods fail to consider the rich temporal information present in video sequences. Existing video iTM methods often still treat frames in isolation, neglecting the crucial inter-frame dependencies that are vital for smooth and coherent HDR video.
Methods
The proposed KPN-MFI framework builds upon a kernel prediction network (KPN) architecture, which is known for its effectiveness in image iTM. The core innovation lies in the introduction of a Multi-Frame Interaction (MFI) module.
Here's a breakdown of the key components:
- Encoder-Decoder KPN Backbone: This part of the network is responsible for processing individual frames. It extracts features at different levels. A "long skip connection" is used to pass low-level features directly to the decoder, aiding in the reconstruction of fine-grained details.
- Multi-Frame Interaction (MFI) Module: This is the heart of the video processing. It operates on the high-level features extracted by the encoder. The MFI module aims to capture spatial-temporal context by explicitly modeling the relationships between adjacent frames (e.g.,
St-1,St,St+1). It uses a combination of cosine distance and channel attention mechanisms to highlight relevant information across frames. This module can be integrated into existing image-based KPN architectures to enable video processing. - Adaptive Pixel-wise Convolution (PC): Instead of directly predicting HDR pixel values, the KPN predicts pixel-wise kernels. These kernels are then applied to the input SDR frame to generate the HDR output. This approach, as demonstrated in prior work, helps to alleviate artifacts in under- or over-exposed regions.
- Brightness Consistency Loss: To further enforce temporal coherence, a novel brightness consistency loss function is introduced. This loss is based on a Gaussian pyramid decomposition and uses optical flow to align adjacent frames before calculating the consistency. This approach is designed to be more robust to potential misalignments compared to methods relying solely on optical flow.
Datasets
The authors used a dataset consisting of 21 pairs of 4K-UHD HDR videos in the HDR10 standard, along with their SDR counterparts, sourced from YouTube.
- 17 video pairs were used for training.
- 4 video pairs were used for testing.
- Frames were sampled at intervals of two seconds, with scene transitions removed.
- The dataset was further augmented by cropping frames to 1024x1024, resulting in 3,241 training pairs and 482 testing pairs.
Key Insights and Ideas
- Leveraging Temporal Context is Crucial for Video iTM: The paper strongly emphasizes that simply processing frames independently is insufficient for high-quality video HDR conversion. The MFI module's success highlights the importance of explicitly modeling inter-frame dependencies.
- Modularity of the MFI Module: A significant contribution is the design of the MFI module as a plug-and-play component. This allows existing state-of-the-art image iTM methods to be readily extended to video iTM by simply incorporating this module, lowering the barrier to entry for video HDR research.
- Adaptive Kernels for Artifact Reduction: The use of pixel-wise kernel prediction, rather than direct pixel value prediction, is shown to be effective in mitigating common artifacts in HDR reconstruction.
- Robust Brightness Consistency Loss: The proposed Gaussian pyramid-based brightness consistency loss offers a more resilient approach to ensuring temporal smoothness, especially in complex scenes where optical flow might be less reliable.
Key Results
The KPN-MFI model demonstrated superior performance across various metrics:
- Quantitative: The method achieved state-of-the-art results on PSNR, ΔEITP, and HDR-VDP3, indicating better overall image quality, perceptual similarity, and temporal consistency. For instance, it achieved a PSNR of 37.7260 and an ΔEITP of 9.1975 on the test set.
- Qualitative: Visual comparisons showed that KPN-MFI significantly reduced artifacts such as blurring, over-sharpening, false contours, and incorrect colors, producing more visually pleasing HDR video outputs.
- Efficiency: The model strikes a good balance between performance and computational cost, with a relatively low parameter count and GFLOPs compared to some other methods.
Limitations and Future Work
- Limitations: While the paper demonstrates strong results, the authors acknowledge that the effectiveness of optical flow for frame alignment in their brightness consistency loss can be limited in very complex scenes.
- Future Work: The authors suggest that the KPN-MFI model's architecture, particularly the MFI module, could be generalized to other low-level vision tasks that involve processing sequential data, such as video super-resolution or video denoising. Further exploration in these areas is a promising direction.
Summary updated