TL;DR
This paper tackles the challenge of super-resolving (SR) images from noisy, unpaired datasets, a common issue in real-world applications like semiconductor inspection. They propose a practical method involving averaging multiple noisy images to reduce noise before SR, and demonstrate its effectiveness in improving image quality and detail preservation.
What Problem Does This Paper Address?
Super-resolution (SR) aims to enhance the resolution of low-resolution (LR) images to produce high-resolution (HR) ones, crucial for tasks like detailed visual inspection. While supervised SR methods excel with paired, clean datasets, real-world scenarios often present noisy and misaligned LR-HR image pairs. This misalignment and noise make direct application of standard SR techniques problematic, as upsampling can amplify noise and introduce artifacts, while misalignments prevent effective training. Existing SR methods often assume clean or minimally noisy inputs, failing to address the severe noise encountered in practical acquisition settings.
Methods
The core of the proposed methodology lies in average denoising applied to noisy LR images before super-resolution. The authors hypothesize that averaging multiple noisy captures of the same scene can reduce noise while preserving structural details.
- Unpaired Data Generation & Average Denoising:
- They start with a ground-truth HR dataset (public SEM dataset).
- To simulate real-world conditions, they introduce random shifts (misalignments) to the HR images and then downsample them to create clean LR images.
- Significant noise (with variances of 30, 60, and 90) is then injected into these clean LR images to create noisy LR images.
- Crucially, they then take multiple instances of these noisy LR images (e.g., 4, 8, or 16) and average them to create a single "averaged LR" image. This averaged image is then used as the input for SR.
- Paired Data Generation (for training):
- To enable supervised SR training, they create paired HR-LR datasets.
- They use a technique inspired by RealSR [4], which involves extracting noise patches from the averaged LR images and injecting them into the downsampled HR images. This process aims to create paired data that mimics the noise characteristics of the averaged LR images.
The SR task itself is then performed using a standard SR network (SwinIR is used as a baseline) on these generated paired datasets.
Datasets
- Ground-truth HR Dataset: A public SEM dataset [3] was used as the source for ground-truth HR images.
- Synthetically Generated Datasets: The authors generated their own datasets for training and testing, simulating real-world noisy and unpaired conditions. This involved:
- Creating noisy LR images by downsampling HR images, introducing random shifts, and adding significant noise.
- Generating "averaged LR" images by averaging multiple noisy LR images.
- Creating paired HR-LR datasets for supervised training by injecting noise patches into downsampled HR images.
Key Insights and Ideas
- Average Denoising as a Preprocessing Step: The paper strongly advocates for average denoising as a practical and effective preprocessing step for SR in noisy environments, rather than relying solely on advanced SR network architectures.
- Trade-off Management: The authors highlight the critical trade-off between noise reduction and detail preservation. Their average denoising approach aims to strike a better balance than algorithmic denoising methods that might remove details along with noise.
- Scalability of Average Denoising: The effectiveness of average denoising scales with the number of images averaged. More averaging leads to better noise reduction and detail preservation, up to a point where computational cost becomes a factor.
- Unpaired Data is the Norm: The work acknowledges that acquiring perfectly aligned, clean paired datasets for SR is often impractical, especially in specialized fields like semiconductor inspection. Their approach directly addresses this by working with unpaired, noisy data.
Key Results
- Improved Performance with Averaging: The proposed method, using average denoising with an increasing number of averaged images (from 4 to 16), consistently improved SR performance across various metrics (PSNR, SSIM, LPIPS, NIQE).
- Quantitative Gains: For a noise variance of 60, using 16 averaged images showed improvements of 0.68 and 0.0279 in PSNR and SSIM, respectively, compared to using 4 averaged images. Further gains were observed in LPIPS and NIQE.
- Qualitative Improvements: Visual results demonstrate that averaging significantly reduces noise and restores structural details and texture that are lost in single noisy LR images or images processed with fewer averages. This is particularly evident at higher noise levels (variance 60 and 90).
- Effectiveness Across Noise Levels: The method proved effective across different noise levels (variances 30, 60, and 90), with the most significant benefits seen in the highest noise scenarios.
Limitations and Future Work
- Manual Noise Extraction: The paper notes that noise extraction for generating paired data needs to be carefully and manually performed. This is critical for accurately capturing the source domain image characteristics.
- Severely Corrupted Images: While effective for high noise levels, the scheme struggled with images that were "indistinguishable noise and details," making it nearly impossible to extract meaningful noise patches.
- Future Work: The authors suggest that their average denoising concept can be applied to other general SR models and even transform-based networks, indicating potential for broader applicability. They also mention that for different ground-truth datasets, the proposed scheme can be applied to train SR models.
Summary updated