Abstract
Traditional style transfer methods often treat images as global feature collections, leading to "content mismatch" where stylistic textures spill across unrelated semantic boundaries. We introduce DeepObjStyle, a training-independent framework that utilizes object-level supervision to preserve the semantic structure of content images. By employing a novel "unmapped object loss," our method gracefully handles cases where content and style images contain different numbers or types of objects. Experimental results, including NIMA and Pieapp metrics alongside a large-scale user study, demonstrate that DeepObjStyle significantly outperforms existing global style transfer methods in both image quality and structural fidelity.
Introduction
Classic style transfer techniques, such as the Gatys et al. approach, frequently fail when the content and style images are semantically disparate. For instance, transferring a "cityscape" style onto a "portrait" often results in textures like windows or bricks being mapped onto human features, a phenomenon known as style spillover. This problem is particularly acute when there is a mismatch in the number of objects present in each image. DeepObjStyle addresses these challenges by shifting the focus from global texture matching to object-aware stylistic supervision.
Methodology
DeepObjStyle is a training-independent optimization framework that utilizes features from a pre-trained VGG19 network. The core of the system is an object-based mapping strategy:
- Object Mapping (OM): Segmentation masks are used to identify contextually similar objects between the style ($S$) and content ($C$) images.
- Mapped Object Loss ($\mathcal{L}_{dos}^M$): For paired objects (e.g., sky to sky), the system combines deep photo-style loss with contextual content loss to ensure realistic results.
- Unmapped Object Loss ($\mathcal{L}_{dos}^U$): This component handles mismatches through two strategies:
- Style Diffusion (STP-C): When the content has more objects, style features are diffused across the additional content regions.
- Style Utilization (STP-S): When the style has more objects, extra features are utilized without compromising the content's geometric structure.
Experimental Results
The effectiveness of DeepObjStyle was validated through quantitative benchmarks and qualitative comparisons against state-of-the-art models like WCT2.
Quantitative Performance
In evaluations using 100 diverse style transfer instances, DeepObjStyle achieved a NIMA (Image Quality) score of 5.49, surpassing WCT2 (5.35) and STROTSS (4.88). It also recorded the lowest Pieapp (Perceptual Error) score of 2.83, indicating high faithfulness to the original content structure. A user study involving 60 experts further confirmed these findings, with DeepObjStyle receiving 680 votes compared to WCT2's 174.
Structural Preservation
A unique "Word Cloud Test" was employed to assess geometric distortion. While competing methods often rendered text illegible, DeepObjStyle successfully preserved the readability of the word clouds, proving its ability to maintain underlying geometric structures during the transfer process.
Conclusion
DeepObjStyle provides a robust solution to the content mismatch problem in photo style transfer. By prioritizing object-level boundaries and introducing strategies for unmapped objects, it produces results that are both stylistically rich and semantically accurate. While the method is optimized for realism and may occasionally miss fine-grained artistic textures, it sets a new standard for structural preservation in the field.
Summary updated