Abstract
This paper presents a novel contribution to the field of regional style transfer. Existing methods often suffer from the drawback of applying style homogeneously across the entire image, leading to stylistic inconsistencies or foreground object distortion when applied to images with foreground elements such as person figures. To address this limitation, we propose a new approach that leverages a segmentation network to precisely isolate foreground objects within the input image. Subsequently, style transfer is applied exclusively to the background region. The isolated foreground objects are then carefully reintegrated into the style-transferred background. To enhance the visual coherence between foreground and background, a color transfer step is employed on the foreground elements prior to their re-incorporation. Finally, we utilize feathering techniques to achieve a seamless amalgamation of foreground and background, resulting in a visually unified and aesthetically pleasing final composition. Extensive evaluations demonstrate that our proposed approach yields significantly more natural stylistic transformations compared to conventional methods.
Introduction
Neural Style Transfer (NST) has revolutionized artistic image generation, yet it faces significant hurdles when processing images with distinct foreground subjects. Global application of style often results in the "twisting" of human features, where facial structures and body forms are lost to the textures of the style reference. This paper addresses the need for visual coherence—maintaining the structural integrity of the subject while ensuring they appear naturally integrated into a stylized environment. The challenge lies in achieving a middle ground: stylizing the environment while harmonizing the subject's colors to match the new atmosphere, avoiding the "flat sticker" look of simple background-only transfers.
Methodology
The proposed framework utilizes a multi-stage pipeline to decouple style and color, ensuring that the subject remains recognizable while the scene feels unified.
- Semantic Segmentation: The system uses DeepLabv3+ (with a ResNet-101 backbone) to generate a precise mask of the foreground object (specifically "person" in this study).
- Boundary Optimization: Raw segmentation masks are often noisy. The authors apply Canny edge detection and a Breadth-First Search (BFS) algorithm to refine the edges. They then use Euclidean distance-based smoothing to create a "feathered" edge, which is critical for natural blending later.
- Artistic Style Transfer: Using a "fast style transfer" approach, the model applies the artistic style exclusively to the background region. This preserves the original content of the background while adopting the textures and colors of the style reference.
- Foreground Color Transfer: Instead of stylizing the person, they perform color transfer in the $l\alpha\beta$ color space. By using Principal Component Analysis (PCA) to map the color distribution of the style image onto the foreground subject, the person adopts the "mood" and lighting of the artwork without losing their physical features.
- Alpha Blending: Finally, the stylized background and the color-harmonized foreground are merged using the feathered mask to ensure there are no harsh "halos" or jagged artifacts at the borders.
Results and Discussion
The segmentation component was trained and evaluated on the PASCAL VOC 2012 dataset, achieving a mean Intersection-over-Union (mIoU) of 82.1% on the validation set.
Qualitatively, the regional approach preserves the human subject's identity far better than global methods, which often distort the person beyond recognition. As shown in the comparative results, the proposed method also provides superior integration compared to simple background-only replacement, which often looks disjointed. By using the $l\alpha\beta$ space for color transfer, the method achieves a "mood" match that mimics human visual perception, making the subject appear realistically lit by the colors of the painting behind them. Furthermore, the use of a fast style transfer architecture avoids the heavy computational cost of traditional optimization-based NST, making the pipeline more viable for practical applications.
Conclusion
This research demonstrates that regional processing is essential for high-quality artistic transformation of images containing prominent foreground subjects. By decoupling style and color, the framework maintains subject integrity while achieving environmental harmony. Future work will focus on expanding the object classes beyond "person," optimizing for high-resolution inference, and addressing temporal consistency for video applications to prevent flickering between frames.
Summary updated