Abstract
Single Image Super-Resolution (SISR) is a fundamentally ill-posed problem where multiple high-resolution (HR) images can correspond to a single low-resolution (LR) input. While deep learning has significantly advanced the field, models often exhibit high uncertainty in regions with complex textures and sharp edges. We propose SSC-SR (Self-Supervised Constraint for Super-Resolution), a "plug-and-play" framework designed to enhance existing SR models by introducing self-supervised consistency constraints. By utilizing a dual-network architecture where a target network is updated via Exponential Moving Average (EMA), the framework stabilizes learning and improves reconstruction in challenging image regions. Our experiments demonstrate consistent performance gains across various architectures, including CNNs and Transformers, particularly on benchmarks featuring repetitive man-made structures.
Introduction
The primary challenge in SISR is the inherent ambiguity in reconstructing fine details from degraded inputs. While modern architectures like Transformers have pushed performance boundaries, they still struggle with pixel-level uncertainty in high-frequency areas. Furthermore, traditional self-supervised methods often rely on contrastive learning with negative samples, which is difficult to adapt for the precise reconstruction requirements of low-level vision tasks.
SSC-SR addresses these issues by drawing inspiration from the Bootstrap Your Own Latent (BYOL) philosophy. Instead of relying on negative samples, it enforces consistency between different augmented views of the same image through an asymmetric dual-network setup. This strategy allows the model to extract more robust features from standard training data without requiring additional labels.
Methodology
The SSC-SR framework is built around a dual asymmetric architecture designed to prevent solution collapse while providing a stable learning signal.
Dual Asymmetric Architecture
The system consists of two branches:
- Online Network ($f_{SR}$): The primary model undergoing active training.
- Target Network ($\hat{f}_{SR}$): A structural twin of the online network, but its weights are updated as an Exponential Moving Average (EMA) of the online network. This acts as a stabilized "teacher" that provides a consistent target for the online branch.
To prevent the networks from converging to a trivial identity mapping, an asymmetric Projection Head (a small set of convolutional layers) is added exclusively to the online branch.
Training Objectives
During training, an LR image $I^{LR}$ is subjected to two different data augmentations (e.g., rotations $R_1, R_2$). The online network processes one version, while the target network processes the other. The model is optimized using a composite loss function:
- Supervised Reconstruction Loss ($\mathcal{L}_r$): Ensures the output matches the ground-truth HR image.
- Self-Supervised Consistency Loss ($\mathcal{L}_c$): Penalizes discrepancies between the outputs of the online and target branches after accounting for the geometric transformations. This forces the model to be invariant to augmentations and more certain in its reconstructions.
Experiments and Results
We evaluated SSC-SR by integrating it into several state-of-the-art base models, including EDSR, RCAN, NLSN, SwinIR, and HAT. Training was conducted on the DIV2K and DF2K datasets, with testing performed on standard benchmarks: Set5, Set14, B100, Urban100, and Manga109.
Quantitative Performance
SSC-SR consistently improved the PSNR (Peak Signal-to-Noise Ratio) of every base model. For instance, it achieved an average increase of 0.1 dB over the vanilla EDSR and 0.06 dB over SwinIR. The most significant gains were observed on the Urban100 and Manga109 datasets, which are rich in sharp edges and repetitive patterns. On Manga109, EDSR saw a substantial improvement of 0.20 dB.
Qualitative Analysis
Visual results confirm that models trained with SSC-SR produce sharper edges and more accurate fine textures. As shown in the comparison figures, the "Ours" versions of standard models effectively reduce aliasing artifacts and blurriness in complex regions like building facades, animal fur, and fine line art.
Discussion and Conclusion
SSC-SR provides a "free lunch" for super-resolution by repurposing standard data augmentations into a self-supervised signal. By harnessing the divergence between the online model and its historical average, the framework identifies and corrects regions of high uncertainty.
While the benefits are slightly less pronounced on massive datasets where models are already near-optimal, the framework's plug-and-play nature makes it a valuable tool for boosting any SR architecture. Future work will explore extending this consistency paradigm to other low-level tasks such as image denoising and deraining, where handling complex local uncertainty remains a critical challenge.
Summary updated