Abstract
Recent studies have significantly enhanced the performance of single-image super-resolution (SR) using convolutional neural networks (CNNs). While there can be many high-resolution (HR) solutions for a given input, most existing CNN-based methods do not explore alternative solutions during the inference. A typical approach to obtaining alternative SR results is to train multiple SR models with different loss weightings and exploit the combination of these models. Instead of using multiple models, we present a more efficient method to train a single adjustable SR model on various combinations of losses by taking advantage of multi-task learning. Specifically, we optimize an SR model with a conditional objective during training, where the objective is a weighted sum of multiple perceptual losses at different feature levels. The weights vary according to given conditions, and the set of weights is defined as a style controller. Also, we present an architecture appropriate for this training scheme, which is the Residual-in-Residual Dense Block equipped with spatial feature transformation layers. At the inference phase, our trained model can generate locally different outputs conditioned on the style control map. Extensive experiments show that the proposed SR model produces various desirable reconstructions without artifacts and yields comparable quantitative performance to state-of-the-art SR methods.
Introduction
Single-image super-resolution (SISR) is a fundamentally ill-posed problem, as a single low-resolution (LR) input can correspond to an infinite number of high-resolution (HR) possibilities. Traditionally, practitioners have had to choose between two extremes: Mean Squared Error (MSE)-based models, which optimize for pixel accuracy but result in blurry outputs, and Generative Adversarial Network (GAN) or perceptual models, which produce sharp textures but often introduce "hallucinated" artifacts or distort fine details.
Existing methods to bridge this gap typically require training multiple separate models and interpolating their weights or outputs. This approach is computationally expensive and generally applies a global style change to the entire image. The proposed FxSR (Flexible Style Super-Resolution) framework addresses these limitations by enabling a single model to generate a wide spectrum of HR outputs—from smooth to texture-heavy—at inference time with local control.
Methodology
FxSR treats the style of super-resolution as a multi-task learning problem. The core innovation is a Conditional Objective, which is a weighted sum of various losses, including pixel-wise reconstruction, adversarial, and perceptual losses at different VGG feature levels.
Key Components
- Style Controller ($t$): A parameter map that adjusts the weights of the loss functions. For instance, $t=0$ might prioritize reconstruction accuracy (low blur), while $t=1$ favors perceptual sharpness.
- Architecture (RRDB + SFT): The model utilizes Residual-in-Residual Dense Blocks (RRDB) equipped with Spatial Feature Transform (SFT) layers. These layers take the style map as input to modulate internal features, allowing the network to adapt its behavior spatially across the image.
- Training Strategy: The model is trained using various randomly sampled style maps, learning to associate specific parameter values with distinct visual styles. This allows for two primary variants: FxSR-PD (focusing on the Perception-Distortion trade-off) and FxSR-DS (focusing on Diversity by using different VGG feature levels).
Experiments and Results
The models were trained on the DIV2K dataset and evaluated on standard benchmarks including BSDS100 and General100.
Qualitative and Quantitative Performance
FxSR-PD achieves state-of-the-art results on the Perception-Distortion (P-D) plane. At $t=0$, it yields high fidelity (PSNR), while at $t=0.8$, it achieves perceptual quality (LPIPS) comparable to or better than ESRGAN. Visually, FxSR avoids the "overshoot" artifacts (white halos) common in other perceptual models and produces more natural textures.
Local Control and Flexibility
A significant advantage of FxSR is the ability to apply style maps locally. By using a mask, a user can set $t=1$ for a textured subject in the foreground and $t=0$ for a smooth background. This prevents the model from adding sharp noise to naturally blurry areas, effectively mimicking a photographic "bokeh" effect when combined with depth maps.
Conclusion
FxSR provides a highly efficient and flexible framework for super-resolution, replacing multiple specialized models with a single, adjustable architecture. By leveraging conditional objectives and SFT layers, it offers unprecedented local control over image style. Future work may extend this conditional framework to other restoration tasks such as denoising and deblurring, where local adaptation is equally vital.
Summary updated