Editorial: Introduction to the Issue on Deep Learning for Image/Video Restoration and Compression

A. Murat Tekalp, Michele Covell, Radu Timofte, Chao DongIEEE JSTSP 2021 · DOI · OpenAlex

Introduction

The field of image and video processing is undergoing a fundamental paradigm shift. For decades, signal processing relied on linear mathematical models to address degradation. However, the rise of deep learning (DL) has introduced a powerful nonlinear framework that is redefining the state of the art. This editorial synthesizes the contributions of a special issue of the IEEE Journal of Selected Topics in Signal Processing, highlighting how neural networks are moving beyond laboratory settings to tackle the complex, high-stakes challenges of real-world restoration and data compression.

The Limitations of Traditional Signal Processing

Traditional methods often assume known degradation models, such as Gaussian noise or bicubic downsampling. While mathematically elegant, these models fail to capture the complexity of real-world artifacts—such as the specific blur of a smartphone lens or the distortions caused by atmospheric turbulence. The current research landscape identifies three critical gaps:

  1. Generalization: Models trained on synthetic data often fail when applied to "in-the-wild" images.
  2. Efficiency: High-performing architectures are frequently too computationally intensive for real-time use on mobile or edge devices.
  3. Perceptual Optimization: Standard metrics like Peak Signal-to-Noise Ratio (PSNR) often favor "safe," blurry outputs over images that appear sharp and realistic to human observers.

Methodological Advancements

The papers within this special issue explore several innovative methodologies to bridge these gaps:

Blind Restoration and Super-Resolution (SR)

Researchers are moving away from fixed degradation assumptions toward "blind" restoration. This includes Zero-Shot SR, which utilizes internal image statistics, and the use of Generative Adversarial Networks (GANs) to model complex, unknown degradations. Supervised learning on paired "real-world" datasets has also become a cornerstone for training models that generalize better to actual camera output.

Neural Architecture Search (NAS) and Efficiency

To address the efficiency gap, automated Neural Architecture Search (NAS) is being employed to discover optimized network structures, such as U-Net variants, that maintain high performance while adhering to the hardware constraints of consumer electronics.

End-to-End and Hybrid Compression

The traditional codec pipeline (e.g., HEVC/H.265) is being challenged by deep autoencoders that optimize jointly for bit-rate and visual distortion. Furthermore, "hybrid" models are emerging, where CNN-based in-loop filters are integrated into existing video standards to provide immediate performance gains.

3D Point Cloud Processing

Beyond 2D frames, Graph Convolutional Networks (GCNs) are being applied to 3D Point Clouds (PCs). These networks handle the irregular data structures inherent in 3D geometry, offering new solutions for denoising and geometry coding.

The Role of Real-World Datasets

A significant trend identified in this issue is the transition from synthetic to real-world datasets. Pivotal datasets such as DPED, RealSR, Zurich RAW-to-RGB, and DRealSR provide the necessary ground truth for training models on actual hardware degradations. Additionally, the PIPAL dataset for Image Quality Assessment (IQA) helps bridge the gap between algorithmic scoring and human visual preference.

Key Insights: The Perception-Distortion Tradeoff

One of the most critical theoretical insights discussed is the mathematical limit known as the perception-distortion tradeoff. This principle states that as a model's fidelity to the original pixels (distortion) increases, its perceived realism (perceptual quality) may actually decrease. Models like ESRGAN and RankSRGAN are specifically engineered to navigate this tension, producing results that look "real" even if they deviate slightly from the pixel-perfect ground truth.

Landmark Results

The special issue reports several significant benchmarks:

  • Compression Efficiency: The Video-based Point Cloud Compression (V-PCC) reference model has demonstrated compression ratios as high as 125:1 while maintaining acceptable quality.
  • Video Performance: End-to-end learned models for low-latency video compression are now outperforming established standards like H.264 and HEVC in specific metrics such as MS-SSIM.
  • Super-Resolution: Models like ESRGAN have set new standards in the PIRM challenge, significantly improving texture detail compared to previous generative approaches.

Future Directions and Challenges

Despite these gains, several hurdles remain for the next generation of researchers:

  • Interpretability: The "black box" nature of deep models makes it difficult to guarantee performance bounds or understand specific failure modes.
  • Temporal Consistency: In video processing, preventing "flicker" between frames remains a computationally expensive challenge.
  • Edge Deployment: Shrinking models with millions of parameters for real-time inference on mobile devices is essential for commercial viability.

The transition from linear signal processing to deep learning is not merely an incremental improvement but a fundamental redefinition of how we capture, store, and restore visual information.

Summary updated

Abstract

The papers in this special issue focus on deep learning for image/video restoration and compression. The huge success of deep-learning-based approaches in computer vision has inspired research in learned solutions to classic image/video processing problems, such as denoising, deblurring, dehazing, deraining, super-resolution (SR), and compression. Hence, learning-based methods have emerged as a promising nonlinear signal-processing framework for image/ video restoration and compression. Recent works have shown that learned models can achieve significant performance gains, especially in terms of perceptual quality measures, over traditional methods. Hence, the state of the art in image restoration and compression is getting redefined. This special issue covers the state of the art in learned image/video restoration and compression to promote further progress in innovative architectures and training methods for effective and efficient networks for image/video restoration and compression.
BibTeX
@article{tekalp2021editorial,
  author = {A. Murat Tekalp and Michele Covell and Radu Timofte and Chao Dong},
  title = {{Editorial: Introduction to the Issue on Deep Learning for Image/Video Restoration and Compression}},
  journal = {IEEE Journal of Selected Topics in Signal Processing},
  year = {2021},
  doi = {10.1109/jstsp.2021.3053364},
}

Loading related content...

Figure