Introduction
The rapid evolution of video acquisition and processing technologies—marked by higher resolutions and sophisticated compression standards like H.265/HEVC—has created a "moving target" for enhancement and quality assessment. While deep learning has significantly advanced computer vision, traditional algorithms often struggle with the complexities of real-world video. This special section in the IEEE Transactions on Circuits and Systems for Video Technology (November 2020) presents ten high-impact papers that define the "new frontier" of the field, emphasizing the transition from synthetic laboratory models to the handling of "authentic" distortions.
The Challenge of Authentic Distortions
A critical gap exists in current video processing research: many models perform exceptionally well on synthetic data, such as Gaussian noise, but fail when confronted with real-world artifacts. These "authentic" distortions include physical camera blur, complex underwater light scattering, and the intricate patterns of natural rain. The research presented here aims to bridge this gap by developing models that incorporate the underlying physics and perceptual nuances of real-world environments.
Methodological Advancements
The featured papers introduce diverse deep learning architectures tailored for specific enhancement tasks:
- Discrepancy Learning for Blur Assessment: A novel approach generates "discrepancy maps" between distorted and restored video versions, extracting sparse representation entropy to assess realistic blur.
- Dual-Domain Processing (IDCN): The Implicit Dual-Domain Convolutional Network cleans compression artifacts by simultaneously leveraging pixel-domain features and Discrete Cosine Transform (DCT) frequency-domain priors.
- Cascaded Spatial Feature Modulation (CSFM): This network for super-resolution utilizes multilevel information through channel-wise and spatial modulation.
- Internal and External Correlation (IENet): A denoising network that enhances performance by searching for similar patches both within the target image and across external web-guided references.
- Conditional GANs (CGAN) for De-raining: A multiscale discriminator architecture is employed to ensure both visual realism and quantitative accuracy in single-image de-raining.
Emerging Benchmarks
To support the development of these models, several new datasets have been introduced to the research community:
- IVIPC-DQA: A database for de-raining quality assessment containing 206 authentic rain images and their de-rained counterparts.
- PEA265: A comprehensive, subject-labeled database for H.265/HEVC compressed video, featuring over 60,000 labels across six Perceptual Evaluation of Artifacts (PEA) categories.
- Lens Distortion Datasets: New synthesized datasets designed to train models for rectifying distortions in dynamic sequences.
Discussion: Perceptual and Frequency-Domain Considerations
The research highlights several key shifts in the field:
- Authenticity as a Priority: The focus on "authentic" rain and "realistic" blur indicates a maturation of the field beyond "toy problems" involving synthetic noise.
- The Role of Perceptual Metrics: There is a growing emphasis on evaluation. Models like the PEA265-based Video Quality Model (VQM) aim to "see" artifacts as a human observer would, which is vital for next-generation codecs.
- Frequency Domain Integration: The success of dual-domain models like IDCN underscores the importance of combining spatial CNN features with frequency-domain information, where video compression primarily occurs.
- Motion Handling: For frame interpolation, multiscale structures (e.g., MS-PFT) are essential for managing "ghosting" artifacts across varying motion speeds.
Experimental Performance and Results
The proposed methods demonstrate significant improvements over existing standards:
- Realistic Blur Assessment (RBA): This method outperformed both general-purpose and specialized quality metrics on real-world databases.
- De-raining (B-FEN): The Bidirectional Feature Embedding Network showed superior performance compared to established No-Reference Image Quality Assessment (NR-IQA) models.
- Compression Artifact Reduction: The PEA-based video quality model achieved performance parity with industry standards while offering deeper insights into specific artifact types.
- Underwater Correction: Unsupervised adaptation networks proved effective for depth estimation and color correction without requiring ground-truth "clean" images.
Conclusion and Future Directions
While these advancements represent a significant leap forward, challenges remain. Handling extreme occlusion and large-scale motion in video interpolation continues to be difficult. Furthermore, while unsupervised methods show promise, many tasks still rely on expensive, large-scale labeled datasets. Finally, the computational overhead of deep CNN and GAN architectures remains a hurdle for real-time applications, marking the next frontier for research in video enhancement.
Summary updated