Abstract
A dynamic texture (DT) refers to a sequence of images that exhibit temporal regularities and has many applications in computer vision and graphics. Given an exemplar of dynamic texture, it is a dynamic but challenging task to generate new samples with high quality that are perceptually similar to the input exemplar, which is known to be example-based dynamic texture synthesis (EDTS). Numerous approaches have been devoted to this problem, in the past decades, but none them are able to tackle all kinds of dynamic textures equally well. In this paper, we investigate the synthesizability of dynamic texture samples: given a dynamic texture sample, how synthesizable it is by using EDTS, and which EDTS method is the most suitable to synthesize it? To this end, we propose to learn regression models to connect dynamic texture samples with synthesizability scores, with the help of a compiled dynamic texture dataset annotated in terms of synthesizability. More precisely, we first define the synthesizability of DT samples and characterize them by a set of spatiotemporal features. Based on these features and an annotated dynamic texture dataset, we then train regression models to predict the synthesizability scores of texture samples and learn classifiers to select the most suitable EDTS methods. We further complete the selection, partition and synthesizability prediction of dynamic texture samples in a hierarchical scheme. We finally apply the learned synthesizability to detecting synthesizable regions in videos. The experiments demonstrate that our method can effectively learn and predict the synthesizability of DT samples.
Introduction
Dynamic textures (DTs)—videos of phenomena like flowing water, flickering fire, or swaying trees—are notoriously difficult to synthesize. While dozens of Example-based Dynamic Texture Synthesis (EDTS) methods exist, they all have "blind spots." Some handle temporal flow well but fail at spatial patterns; others produce repetitive, "verbatim" artifacts.
This research shifts the focus from building better algorithms to understanding the input data itself. The authors identify a major gap: the lack of a quantitative way to measure a video's "synthesizability" before processing. This paper aims to automate algorithm selection by answering how likely any current method is to produce a high-quality result and which specific algorithm is the case-optimal choice for a particular sample.
Methodology
The authors propose a hierarchical pipeline to analyze videos and predict their synthesis potential. The process begins with DT Retrieval, distinguishing dynamic textures from other motions like activities or events using a C3D descriptor.
Once identified as a texture, the system performs Spatial Mode Partitioning. It classifies the DT into two categories: Spatially Homogeneous DTs (SHDT), which are stationary in both space and time (e.g., a field of grass), and Time-stationary DTs (TDT), which only show regularity over time (e.g., a waterfall or fire).
A key technical contribution is the SCOP-DT Descriptor (Shape-based Co-occurrence Patterns for Dynamic Textures). It extends 2D static texture analysis to 3D by using a "Tree of Shapes" for each frame, implicitly encoding temporal consistency. This allows the model to capture both geometric structures and motion patterns simultaneously to predict a synthesizability score $s \in [0, 1]$.
Experimental Results and Insights
The models were trained on a substantial new dataset of 1,729 DT samples aggregated from benchmarks like UCLA and DynTex, manually annotated by experts on a 3-level scale (Good, Acceptable, Bad) based on the results of six representative synthesis algorithms.
Key findings include:
- Spatial vs. Temporal Difficulty: Spatial synthesizability is significantly harder to predict and achieve than temporal synthesizability. Most DTs have strong self-correlation over time, but complex 2D spatial structures remain challenging for algorithms to replicate.
- High Accuracy: The system achieved a 96.23% average precision in identifying DTs and 96.06% accuracy in distinguishing between SHDT and TDT.
- Method Recommendation: The system successfully suggested the "best" synthesis method with an accuracy of 75.76% for SHDTs and 82.39% for TDTs.
Conclusion
By treating synthesizability as a learnable property, this work enables automated video processing pipelines. The authors demonstrated a practical application where the tool automatically "trims" unconstrained videos to find the most synthesizable rectangular sub-regions, significantly improving final synthesis quality. Future work will explore more objective, automated quality metrics to further reduce reliance on human expert annotation.
Summary updated