State of the "Art": A Taxonomy of Artistic Stylization Techniques for Images and Video

Jan Eric Kyprianidis, John P. Collomosse, Tinghuai Wang, Tobias Isenberg

TL;DR

This paper provides a comprehensive survey and taxonomy of Non-Photorealistic Rendering (NPR) techniques for stylizing images and videos. It organizes decades of research into a structured framework, highlighting key trends like the integration of computer vision and the evolution of filtering approaches, offering a valuable resource for understanding the field's past, present, and future.

What Problem Does This Paper Address?

The field of Non-Photorealistic Rendering (NPR), specifically artistic stylization of 2D content (images and video), has seen rapid development over two decades. This has led to a proliferation of techniques, making it challenging for researchers and practitioners to navigate the landscape, understand the relationships between different methods, and identify emerging trends. The paper addresses this by providing a structured taxonomy and historical overview of these techniques, aiming to bring clarity and coherence to the field.

Methods

The paper's primary contribution is a taxonomy that categorizes 2D NPR algorithms based on their design characteristics and behavior. This taxonomy is not based on output styles (like "painterly" or "sketch") but rather on the underlying rendering primitives (strokes, regions, tiles) and the algorithmic approach (local vs. global, image processing vs. computer vision).

The authors also present a chronological overview of the field's development, tracing its evolution from early semi-automatic painting systems to more sophisticated, automated techniques driven by image analysis. They highlight two key complementary trends:

  1. Fusion of Computer Vision and NPR: This involves using higher-level computer vision techniques (like segmentation and scene analysis) to drive artistic abstraction and style diversity.
  2. Evolution of Local Processing: This focuses on the development of edge-aware filtering for real-time stylization, enabling efficient processing of images and video.

The paper also discusses various rendering primitives and algorithmic strategies, including:

  • Stroke-based Rendering (SBR): Emulating brush strokes, with variations in placement (local vs. global) and stroke type (curved, dabs).
  • Region-based Techniques: Using image segmentation to define rendering areas.
  • Mosaicking, Tiling, and Stippling: Approximating images with atomic rendering primitives like tiles or dots.
  • Example-based Rendering (EBR): Learning artistic styles from exemplar images.
  • Image Processing and Filtering: Employing techniques like bilateral filtering and diffusion for stylization.

Datasets

Not discussed in the paper. The paper focuses on surveying existing techniques and their methodologies rather than presenting new experimental results on specific datasets.

Key Insights and Ideas

  • The Power of Taxonomy: Organizing NPR techniques by how they work (primitives, algorithms) rather than what they produce (styles) provides a much clearer understanding of the field's structure and interconnections.
  • From Automation to Interaction: The field has moved from early attempts at full automation to a recent trend of developing interactive tools that empower artists, reflecting the limitations of purely automated approaches and the value of human creativity.
  • Computer Vision as a Driving Force: The integration of advanced computer vision techniques has been crucial for enabling more sophisticated image parsing, leading to greater artistic abstraction and style diversity.
  • Filtering for Real-Time: The evolution of edge-aware filtering techniques has been key to achieving real-time stylization, making NPR practical for video and interactive applications.
  • The "Artistic Turing Test" Remains Relevant: While many traditional styles are now well-covered, the challenge of creating truly novel artistic forms and achieving indistinguishable results from human art remains a significant frontier.

Key Results

This paper is a survey and taxonomy, not an experimental study, so it doesn't present quantitative results in terms of accuracy or speedup. However, its key "results" are the structured insights it provides:

  • A comprehensive taxonomy that categorizes a vast array of NPR techniques.
  • A clear historical narrative of the field's evolution, highlighting key milestones and trends.
  • Identification of major research directions, such as the increasing role of computer vision and the development of interactive tools.
  • A discussion of open challenges, including aesthetic evaluation, interactivity, and the creation of new artistic forms.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Aesthetic Evaluation: A significant challenge remains in objectively evaluating the aesthetic quality of NPR. While progress has been made, a "gold standard" methodology is still lacking.
  • Temporal Coherence in Video: Achieving temporally coherent video stylization without extensive manual intervention is still a difficult compromise.
  • Generality vs. Style Diversity: Filtering methods offer real-time performance but limited style diversity, while vision-based methods offer more diverse styles but often lack generality and robustness.
  • Media Realism: While the paper focuses on rendering primitives, the realism of media emulation (e.g., brush strokes, paint properties) significantly impacts the overall aesthetic and is an active area of research.
  • New Artistic Forms: The paper emphasizes the potential for NPR to create entirely new artistic forms, moving beyond mere imitation of past styles.
  • User Studies and Collaboration: There's a need for closer collaboration with artists and end-users to develop effective tools and understand how users interact with and adapt these technologies.
  • Compositional and Structural Manipulation: Future work could explore manipulating imagery at the composition level, going beyond simple stylization of existing structures.

Summary updated

Abstract

This paper surveys the field of nonphotorealistic rendering (NPR), focusing on techniques for transforming 2D input (images and video) into artistically stylized renderings. We first present a taxonomy of the 2D NPR algorithms developed over the past two decades, structured according to the design characteristics and behavior of each technique. We then describe a chronology of development from the semiautomatic paint systems of the early nineties, through to the automated painterly rendering systems of the late nineties driven by image gradient analysis. Two complementary trends in the NPR literature are then addressed, with reference to our taxonomy. First, the fusion of higher level computer vision and NPR, illustrating the trends toward scene analysis to drive artistic abstraction and diversity of style. Second, the evolution of local processing approaches toward edge-aware filtering for real-time stylization of images and video. The survey then concludes with a discussion of open challenges for 2D NPR identified in recent NPR symposia, including topics such as user and aesthetic evaluation.
BibTeX
@article{kyprianidis2013state,
  author = {Jan Eric Kyprianidis and John P. Collomosse and Tinghuai Wang and Tobias Isenberg},
  title = {{State of the "Art": A Taxonomy of Artistic Stylization Techniques for Images and Video}},
  journal = {IEEE Trans. Vis. Comput. Graph.},
  year = {2013},
  url = {https://openreview.net/forum?id=ZmJNc8Q8xz4},
}
Figure