Introduction
Artistic style transfer has evolved from slow optimization-based processes to fast, feed-forward networks. However, most contemporary methods apply styles globally, transforming the entire image. Achieving localized style transfer—where only specific objects like cars or buildings are styled—has traditionally required computationally expensive instance segmentation and iterative optimization, making real-time video applications impossible.
The CBS Framework
Researchers from the University of British Columbia and Element AI introduced Class-Based Styling (CBS) to bridge this gap. The framework decouples the styling and localization tasks into two parallel branches:
- Fast Semantic Segmentation: Utilizing DABNet (Depth-wise Asymmetric Bottleneck), a lightweight model with only 0.76 million parameters, the system generates a binary mask for a target object class at over 100 FPS.
- Fast Global Styling: Simultaneously, the input frame is processed by a Fast Styling Method (FSM), which generates a globally styled version of the image in a single pass.
The final output is produced by merging these results using the generated mask: $$U = (Mask \times Styled Image) + (Inverse Mask \times Original Image)$$
Experimental Results
The system was evaluated on the CityScapes dataset using high-resolution ($1024 \times 2048$) video frames.
- Speed: CBS achieves a processing rate of 16 FPS on a Tesla P100 GPU. This represents a 250x speedup over previous optimization-based localized styling methods, which typically require 15 seconds per frame.
- Accuracy: DABNet maintains a high Mean IoU of 70.1%, ensuring that the artistic styles are accurately confined to the intended semantic boundaries.
- Flexibility: The modular architecture allows for multi-class styling, where different styles can be applied to different object classes simultaneously by adding more styling branches.
Conclusion and Future Work
CBS demonstrates that real-time localized style transfer is achievable by combining efficient semantic segmentation with feed-forward styling networks. While current results show sharp boundaries, future iterations could incorporate real-time edge-smoothing techniques. The researchers also suggest extending this framework to 3D environments for use in augmented reality and gaming.
Summary updated