TL;DR
CLIPascene is a novel framework for converting photographs into sketches by disentangling abstraction into two controllable axes: fidelity (how closely the sketch follows the image geometry) and simplicity (how many strokes are used). By leveraging the multi-layer semantic hierarchy of CLIP-ViT and a dual-MLP architecture, the system generates vector-based scene sketches that remain recognizable even when reduced to minimal, "loose" representations.
What Problem Does This Paper Address?
Sketching is a fundamental human act of abstraction, yet computational methods often struggle to capture its nuances. Most existing "photo-to-sketch" tools treat abstraction as a single dimension—usually just reducing the number of lines. Furthermore, most models are optimized for single objects (like a lone cat) and fail when faced with complex scenes containing multiple objects, varying backgrounds, and intricate spatial relationships.
The authors identify a lack of control in current methods. A designer might want a sketch that is geometrically precise but sparse, or one that is "loose" and artistic while still being detailed. CLIPascene addresses this by providing a structured way to navigate these different types of abstraction while ensuring the final output is in a flexible vector format (Bezier curves) rather than static pixels.
Methods
CLIPascene does not require a massive pre-existing dataset of sketches. Instead, it performs per-image optimization using two Multi-Layer Perceptron (MLP) networks:
- $MLP_{loc}$ (Stroke Placement): This network learns the optimal parameters (coordinates and shapes) for a set of $n$ Bezier curves to represent the input image.
- $MLP_{simp}$ (Stroke Removal): This network learns a probability for each stroke. During the simplification process, it learns which strokes can be "hidden" (by setting their width to zero) without losing the scene's core meaning.
The Two Axes of Abstraction
- The Fidelity Axis: The authors leverage the fact that different layers of a Vision Transformer (specifically CLIP-ViT) capture different levels of information. Shallow layers focus on edges and geometry (precise fidelity), while deeper layers capture high-level concepts and semantics (loose fidelity). By changing which layer guides the loss function, the model can transition from a literal trace to a conceptual "impression" of the scene.
- The Simplicity Axis: To move from detailed to sparse sketches, the authors introduce a ratio loss. This balances the CLIP similarity (recognizability) against a sparsity penalty. By iteratively tuning this ratio and initializing each level with the previous one, they achieve a perceptually smooth reduction in strokes.
To handle scene complexity, the method uses a "decompose-and-sketch" approach: it separates the foreground from the background using a saliency detector ($U^2$-Net), sketches them independently, and then merges them.
Datasets
Because the model optimizes per-image, it doesn't use a traditional training set. However, for evaluation and benchmarking, the authors curated a set of 35 images across five categories: people, urban, nature, indoor, and animals.
They compared their results against:
- CLIPasso: The state-of-the-art for object sketching.
- UPDG & Chan et al.: GAN-based methods for line drawings.
- Photo-Sketching: A contour-based synthesis method.
Key Insights and Ideas
- Disentanglement is Key: The "aha moment" here is that "abstract" isn't a single setting. A sketch can be abstract because it's messy (low fidelity) or because it's minimalist (high simplicity). Separating these allows for a $4 \times 4$ matrix of artistic choices for every image.
- ViT > ResNet for Scenes: Previous work (CLIPasso) used ResNet-based CLIP. The authors found that ViT-based CLIP is significantly better at capturing the global context of a whole scene, whereas ResNet tends to get lost in local textures.
- Perceptual Smoothness: The authors applied the Weber-Fechner Law—which states that human perception is linear to exponential changes in stimuli—to their simplification process. By using an exponential function to determine stroke removal, the transitions between abstraction levels feel natural to a human viewer.
Key Results
- User Preference: In a study with 25 participants, CLIPascene was preferred over CLIPasso 84.8% of the time for scene sketching. Even against highly detailed GAN methods, users preferred CLIPascene's detailed versions 52.5% of the time.
- Recognizability: Using zero-shot CLIP classification, the authors proved that their sketches remain semantically linked to the original image. Even at high simplicity levels, the top-5 predicted classes for the sketch overlapped significantly with the original photo.
- Designer Utility: Because the output is vector-based, the authors demonstrated that strokes can be easily re-stylized (e.g., changing brush types in Adobe Illustrator) without re-running the model.
Limitations and Future Work
- Computational Cost: Since the model optimizes from scratch for every image, it takes about 6 minutes to generate a sketch on a commercial GPU. This is much slower than feed-forward GANs.
- Dependency on Pre-processing: The quality of the scene decomposition relies heavily on the saliency detector and inpainting models. If the foreground/background split is poor, the final sketch may have artifacts.
- Future Directions: The authors suggest extending the framework to include other axes of abstraction, such as color or specific artistic styles, and optimizing the architecture for faster inference.
Summary updated