TL;DR
The authors introduce FxSR (Flexible Style Super-Resolution), a single model capable of generating a wide spectrum of high-resolution outputs—from smooth, "safe" reconstructions to sharp, texture-heavy ones—at inference time. By using a conditional objective and Spatial Feature Transform (SFT) layers, the model allows users to control the "style" of the reconstruction locally across different regions of a single image.
What Problem Does This Paper Address?
Single-image super-resolution (SISR) is "ill-posed," meaning there are infinitely many high-resolution (HR) images that could correspond to one low-resolution (LR) input. Traditionally, researchers have had to choose their poison:
- MSE-based models: Optimize for pixel accuracy but produce blurry images.
- GAN/Perceptual models: Produce sharp, realistic textures but often introduce "hallucinated" artifacts or distort fine details like text.
Existing solutions to bridge this gap usually involve training multiple separate models and interpolating their weights or outputs. However, these methods are computationally expensive to train and usually apply a global change to the entire image. If you want sharp text but a smooth sky in the same photo, traditional models struggle to give you both simultaneously.
Methods
The researchers treat the style of super-resolution as a multi-task learning problem. Instead of a fixed loss function, they use a Conditional Objective: a weighted sum of various losses (pixel-wise reconstruction, adversarial, and perceptual losses at different VGG feature levels).
Key Components:
- Style Controller ($t$): A parameter (or map) that adjusts the weights of the loss functions. For example, $t=0$ might favor reconstruction accuracy (low blur), while $t=1$ favors perceptual sharpness.
- Architecture (RRDB + SFT): They use the popular Residual-in-Residual Dense Block (RRDB) but equip it with Spatial Feature Transform (SFT) layers. These layers take a "style map" as input and modulate the network's internal features, allowing the model to change its behavior spatially.
- Training Strategy: During training, the model is fed various style maps (randomly sampled) so it learns to associate specific parameter values with specific visual styles.
- Two Flavors:
- FxSR-PD: Focuses on the trade-off between Perception and Distortion.
- FxSR-DS: Focuses on Diversity, using different levels of VGG features to recover different types of textures.
Datasets
The models were trained and evaluated on standard benchmarks in the SR community:
- Training: DIV2K (800 high-quality images).
- Evaluation: BSDS100, General100, and the DIV2K validation set.
- Real-world testing: They also tested on JPEG-compressed images and video frames to demonstrate robustness against compression artifacts.
Key Insights and Ideas
- VGG Layers are Not Created Equal: The authors observed that lower-level VGG features (like ReLU 2-2) are better for sharp edges, while higher-level features (ReLU 4-4) are better for complex textures. FxSR-DS leverages this by allowing the user to "dial in" which feature level to prioritize.
- Local Control is the "Killer Feature": Because the style map is the same size as the LR image, you can feed it a mask. You can set $t=1$ for a textured bird in the foreground and $t=0$ for the out-of-focus background, preventing the model from adding "sharp noise" to naturally blurry areas.
- Efficiency through Conditioning: Rather than storing five different 16.7MB models for five different styles, you store one 18.3MB model that can replicate (and interpolate between) all of them.
Key Results
- Quantitative Performance: FxSR-PD achieves state-of-the-art results on the Perception-Distortion (P-D) plane. At $t=0$, it yields higher PSNR (fidelity) than most perceptual models, and at $t=0.8$, it achieves LPIPS (perceptual quality) scores comparable to or better than ESRGAN.
- Qualitative Superiority: In visual comparisons, FxSR avoids the "overshoot" artifacts (weird white halos around edges) common in ESRGAN and produces more natural textures than SRFlow in many cases.
- Flexibility: The paper demonstrates "Depth-adaptive SR," where a depth map is used as the style map. This automatically makes foreground objects sharper and keeps the background clean, mimicking a natural photographic "bokeh" effect.
Limitations and Future Work
- Style Map Generation: While the model can use local maps, the paper doesn't propose a fully automated way to generate these maps for every image. Users currently have to provide them or use heuristics (like a depth estimator or a text detector).
- Diversity Score: While FxSR produces high-quality diverse styles, its "Diversity Score" (a specific metric used in some challenges) was lower than SRFlow. The authors attribute this to the fact that they used "flat" maps during training rather than more complex distributions.
- Future Directions: The authors suggest expanding this framework to other image restoration tasks like denoising and deblurring, where local control over "how much to fix" is equally critical.
Summary updated