Abstract
We propose a Class-Based Styling method (CBS) that can map different styles for different object classes in real-time. CBS achieves real-time performance by carrying out two steps simultaneously. While a semantic segmentation method is used to obtain the mask of each object class in a video frame, a styling method is used to style that frame globally. Then an object class can be styled by combining the segmentation mask and the styled image. The user can also select multiple styles so that different object classes can have different styles in a single frame. For semantic segmentation, we leverage DABNet that achieves high accuracy, yet only has 0.76 million parameters and runs at 104 FPS. For the style transfer step, we use a popular real-time method proposed by Johnson et al. [7]. We evaluated CBS on a video of the CityScapes dataset and observed high-quality localized style transfer results for different object classes and real-time performance.
Introduction
Style transfer—the process of applying the artistic texture and color of a style image to a content image—has seen significant advancements. Early optimization-based methods produced high-quality results but were computationally expensive, often taking seconds or minutes per frame. While subsequent feed-forward networks enabled real-time global style transfer, they lacked the ability to target specific regions or objects within a scene.
Existing localized styling methods often rely on instance segmentation (e.g., Mask R-CNN) and complex post-processing to ensure temporal consistency and smooth boundaries, which introduces a significant latency bottleneck. This paper introduces Class-Based Styling (CBS), a framework designed to bridge the gap between localization and speed, enabling the styling of specific object classes in high-resolution video at interactive frame rates.
Methods
The core innovation of CBS is its parallel pipeline, which decouples the "where to style" from the "how to style." Instead of a sequential bottleneck, the system processes the input frame through two simultaneous paths:
- Semantic Segmentation: The system utilizes DABNet, an extremely lightweight model with only 0.76 million parameters. It generates binary masks ($R_c$) for specific object classes at speeds up to 104 FPS.
- Global Styling: In parallel, the frame is passed through a Fast Styling Method (FSM) based on the work of Johnson et al., generating a globally styled version of the image ($T_s$).
Blending and Multi-Styling
The final output image ($U$) is synthesized using a simple element-wise blending operation: $$U = (Mask \times Styled\_Image) + (Inverse\_Mask \times Original\_Image)$$
This approach allows for "multi-styling," where different masks can be paired with different styled outputs to apply unique artistic effects to various classes (e.g., cars, buildings, and roads) within the same frame.
Key Results
The CBS framework was evaluated using the CityScapes dataset, a benchmark for urban scene understanding.
- Performance: CBS processes 1024 x 2048 resolution frames at 16 FPS (approximately 60.83 ms per image) on a Tesla P100 GPU. This represents a massive speedup over previous localized methods, which can take up to 15 seconds per frame.
- Accuracy: The DABNet component achieved a 70.1% Mean IoU on the CityScapes test set, ensuring that artistic styles are accurately confined to the intended object boundaries.
- Qualitative Fidelity: The system successfully demonstrates localized styling across diverse classes, maintaining high visual quality even when multiple styles are applied simultaneously.
Limitations and Future Work
While CBS offers unprecedented speed for localized styling, it relies on raw segmentation masks. Unlike slower optimization methods that use Markov Random Fields (MRFs) for boundary smoothing, CBS may occasionally produce sharp edges at the transition between styled and unstyled regions. Additionally, the system is currently limited to the 19 object classes defined in the CityScapes dataset.
Future research will explore extending this localized styling to 3D scenes, potentially enabling real-time artistic augmentation of specific objects in virtual and augmented reality environments.
Summary updated