TL;DR
This paper introduces TCP, a novel generative model for photorealistic image manipulation that allows users to selectively change the style of specific regions within an image without requiring semantic labels. By employing a code consistency loss for disentangling content and style representations and a content alignment loss for preserving background integrity, TCP achieves controllable and flexible region-wise style transfer.
What Problem Does This Paper Address?
Modern image editing tools often struggle to offer both fine-grained control and photorealistic results, especially when it comes to style transfer. While existing methods can perform global style transfer or region-specific edits, they often rely on costly semantic annotations (like object labels) for training and inference, limiting their applicability. Furthermore, disentangling content and style information within an image's latent representation is crucial for achieving consistent and controllable edits, but this has been a persistent challenge. This paper aims to bridge this gap by enabling region-wise style manipulation with high fidelity and controllability, using only self-supervision and user-provided masks.
Methods
The core of the proposed method, named TCP, is an auto-encoder architecture inspired by SwapAuto [37]. An encoder E maps an input image I to a deep feature map, which is then processed by two sub-networks, Hc and Hs, to extract a content code C and a style code S, respectively. The generator G, based on StyleGAN2, then synthesizes an image using these codes.
Two key novel loss functions are introduced:
- Code Consistency Loss (
LCC): This loss enforces an explicit disentanglement between content and style latent representations. It ensures that the content code of a generated image (Cg) is similar to the original content code (CA) and that the style code (Sg) is close to the target style code (SB). This is achieved by feeding a globally style-transferred image back through the encoder and comparing its extracted codes to the original ones. - Content Alignment Loss (
LCA): This loss enables region-wise style transfer. It works by composing the deep feature map of a globally style-transferred image with the feature map of the original content image, guided by a user-provided maskM. This composed feature map is then used by the generator to produce the final image.LCAfurther constrains the synthesized image to ensure that the stylized region aligns with the globally transferred style while the unmasked region remains consistent with the original content.
Crucially, TCP is trained using self-supervision, meaning it does not require explicit semantic labels for training. Randomly generated masks are used during training for LCA, while users provide masks for inference.
Datasets
The model is trained and evaluated on a variety of datasets:
- Training: LSUN-Church [51], Flickr Faces HQ (FFHQ) [19], CelebAMask-HQ [18, 24, 32], Mountains, Waterfalls, Disney faces, and Ukiyoe faces datasets.
- Evaluation: CelebAMask-HQ for region-wise stylization, and FFHQ and LSUN-Church for global image transfer. Cross-domain transfer is evaluated using combinations of FFHQ with Disney faces or Ukiyoe faces.
Key Insights and Ideas
- Self-Supervised Region-wise Editing: The most significant insight is the ability to perform region-wise style transfer without relying on semantic labels. This significantly broadens the applicability of such editing tools.
- Explicit Code Disentanglement: The code consistency loss (
LCC) is crucial for preventing style codes from "leaking" structural information, which was a problem in prior methods like SwapAuto [37]. This leads to more faithful preservation of the original content's structure. - Feature-Level Composition for Region Editing: Composing feature maps rather than directly combining image pixels for region-wise editing helps mitigate boundary artifacts and allows subsequent network layers to smooth out inconsistencies.
- Flexibility of Latent Space: The disentangled latent codes allow for interesting applications like latent space interpolation, enabling gradual transitions between styles or content.
Key Results
- Region-wise Stylization: TCP demonstrates superior content/structure consistency compared to SEAN [58] and SwapAuto [37] on the CelebAMask-HQ dataset. While SEAN can achieve good style transfer, it sometimes loses structure details. SwapAuto struggles with structure consistency due to entangled codes.
- Global Image Transfer: TCP achieves competitive or superior results on FFHQ and LSUN-Church datasets, showing lower FID, SSD, and higher SIFID than most previous methods, indicating better realism and style change.
- Cross-Domain Transfer: The model shows promise for cross-domain style transfer when trained on combined datasets, suggesting a single model could handle multiple domain pairs.
- Qualitative Improvements: Visual results show that TCP effectively preserves background content while stylizing foreground regions, and produces photorealistic outputs with accurate style transfer.
Limitations and Future Work
- Symmetry in Cross-Domain Transfer: The paper notes that performance in cross-domain transfer is not always symmetric for both directions (e.g., A to B vs. B to A).
- Future Directions: The authors plan to explore more flexible and explicit code disentanglement strategies and investigate more advanced applications of their model. They also suggest that the LCA loss could be replaced with other content consistency losses.
Summary updated