TL;DR
This paper introduces PhotoWCT2, a more compact and efficient model for photorealistic style transfer. By employing novel "blockwise training" and "skip connections of high-frequency residuals," PhotoWCT2 achieves state-of-the-art stylization quality with significantly fewer parameters, faster processing, and the ability to handle higher resolution images (like 4K) compared to previous methods.
What Problem Does This Paper Address?
Photorealistic style transfer aims to make an image look like it was created in the style of another image, while still appearing like a real photograph. Existing leading methods, while effective, suffer from being "parameter-heavy." This means they require a lot of computational resources, making them too slow and memory-intensive for high-resolution images (like 4K) and limiting their use on less powerful devices. The paper tackles this by developing a more compact model that maintains high-quality stylization and photorealism.
Methods
The core of PhotoWCT2 lies in two novel mechanisms:
- Blockwise Training: Instead of using a cascade of multiple autoencoders (like PhotoWCT) or a single autoencoder with a less effective feature transformation order (like WCT2), PhotoWCT2 uses a single autoencoder. This is achieved through "blockwise training," where the decoder is trained in stages to learn coarse-to-fine feature transformations. This mimics the strength of cascaded models but within a single, more efficient architecture.
- Skip Connections of High-Frequency Residuals: To ensure that image details are preserved during the transformation process, the paper introduces a new type of skip connection. This method is a simplified and more lightweight version of wavelet-based skip connections (used in WCT2). It focuses on preserving high-frequency information, which is crucial for maintaining image quality and detail, especially when dealing with complex textures and edges. This approach is computationally less expensive than wavelet-based methods.
Datasets
The paper uses the modified version of the DPST dataset for evaluation, which includes 100 stylized images per model. For evaluating image reconstruction capabilities, they sample 500 images from the MSCOCO dataset.
Key Insights and Ideas
- Coarse-to-Fine is Key, but Efficiency Matters: The paper confirms that a coarse-to-fine approach to feature transformation is crucial for strong stylization, but it highlights the inefficiency of previous cascaded implementations.
- High-Frequency Details are Essential for Reconstruction: Preserving high-frequency information is critical for good image reconstruction, and the proposed "skip connections of high-frequency residuals" offer an efficient way to achieve this.
- Blockwise Training Enables Compactness: The blockwise training strategy is a key enabler for creating a single, compact autoencoder that can perform complex, multi-stage feature transformations effectively.
- Trade-offs Can Be Mitigated: The work demonstrates that it's possible to achieve both strong stylization and good image reconstruction without the computational overhead of previous state-of-the-art methods.
Key Results
- Parameter Reduction: PhotoWCT2 uses significantly fewer parameters: 30.3% fewer than WCT2 and 15.6% fewer than PhotoWCT.
- Speed and Resolution: The model is faster than existing methods and is the only one capable of stylizing 4K (UHD) images. For example, it's 0.53 seconds for FHD images, compared to 0.80 seconds for WCT2 and 0.80 seconds for PhotoWCT.
- Quality and Stylization: PhotoWCT2 preserves state-of-the-art image quality and stylization strength, performing comparably to PhotoWCT and WCT2.
- Reconstruction Performance: Ablation studies show that blockwise training combined with high-frequency residuals significantly improves feature and image reconstruction compared to other training strategies and skip connection types.
Limitations and Future Work
The paper acknowledges that while their model is more efficient, it still relies on a pre-trained network (VGGNet) for feature extraction. Future work could explore end-to-end training of the entire network or investigate alternative feature extractors. The authors also mention that their blockwise training approach is demonstrated to be more effective than training from N=4 to N=1, suggesting further exploration into optimal training orders for different network architectures.
Summary updated