Class-Based Styling: Real-Time Localized Style Transfer with Semantic Segmentation

Lironne Kurzman, David Vázquez, Issam H. Laradji

Introduction

Artistic style transfer has evolved from slow optimization-based processes to fast, feed-forward networks. However, most contemporary methods apply styles globally, transforming the entire image. Achieving localized style transfer—where only specific objects like cars or buildings are styled—has traditionally required computationally expensive instance segmentation and iterative optimization, making real-time video applications impossible.

The CBS Framework

Researchers from the University of British Columbia and Element AI introduced Class-Based Styling (CBS) to bridge this gap. The framework decouples the styling and localization tasks into two parallel branches:

  1. Fast Semantic Segmentation: Utilizing DABNet (Depth-wise Asymmetric Bottleneck), a lightweight model with only 0.76 million parameters, the system generates a binary mask for a target object class at over 100 FPS.
  2. Fast Global Styling: Simultaneously, the input frame is processed by a Fast Styling Method (FSM), which generates a globally styled version of the image in a single pass.

The final output is produced by merging these results using the generated mask: $$U = (Mask \times Styled Image) + (Inverse Mask \times Original Image)$$

Comparison between global style transfer and the proposed class-based styling method for 'Cars' and 'Road' classes.

Experimental Results

The system was evaluated on the CityScapes dataset using high-resolution ($1024 \times 2048$) video frames.

  • Speed: CBS achieves a processing rate of 16 FPS on a Tesla P100 GPU. This represents a 250x speedup over previous optimization-based localized styling methods, which typically require 15 seconds per frame.
  • Accuracy: DABNet maintains a high Mean IoU of 70.1%, ensuring that the artistic styles are accurately confined to the intended semantic boundaries.
  • Flexibility: The modular architecture allows for multi-class styling, where different styles can be applied to different object classes simultaneously by adding more styling branches.

Conclusion and Future Work

CBS demonstrates that real-time localized style transfer is achievable by combining efficient semantic segmentation with feed-forward styling networks. While current results show sharp boundaries, future iterations could incorporate real-time edge-smoothing techniques. The researchers also suggest extending this framework to 3D environments for use in augmented reality and gaming.

Summary updated

Abstract

We propose a Class-Based Styling method (CBS) that can map different styles for different object classes in real-time. CBS achieves real-time performance by carrying out two steps simultaneously. While a semantic segmentation method is used to obtain the mask of each object class in a video frame, a styling method is used to style that frame globally. Then an object class can be styled by combining the segmentation mask and the styled image. The user can also select multiple styles so that different object classes can have different styles in a single frame. For semantic segmentation, we leverage DABNet that achieves high accuracy, yet only has 0.76 million parameters and runs at 104 FPS. For the style transfer step, we use a popular real-time method proposed by Johnson et al. [7]. We evaluated CBS on a video of the CityScapes dataset and observed high-quality localized style transfer results for different object classes and real-time performance.
BibTeX
@inproceedings{kurzman2019class,
  author = {Lironne Kurzman and David Vázquez and Issam H. Laradji},
  title = {{Class-Based Styling: Real-Time Localized Style Transfer with Semantic Segmentation}},
  booktitle = {Workshops @ ICCV},
  year = {2019},
  doi = {10.1109/iccvw.2019.00396},
}
Figure