Introduction
The advent of diffusion models like Stable Diffusion has transformed the landscape of artistic image stylization. However, extending these capabilities to video remains a significant hurdle. When frames are processed independently, minor variations in the generation process manifest as "temporal inconsistency"—a phenomenon where the video flickers or "tears" because the applied style does not move cohesively with the underlying scene geometry.
Existing approaches to mitigate this typically involve training large-scale video models, which is computationally expensive, or "zero-shot" modifications to internal attention mechanisms, which often struggle with complex motion. FastBlend introduces a robust, model-free post-processing framework that can be integrated into any image-to-video pipeline to ensure temporal smoothness.
The FastBlend Framework
FastBlend operates entirely in the image space, making it agnostic to the underlying generative model. The pipeline follows a structured three-step process:
- Patch Matching: Using the original "guide" video, the algorithm estimates Nearest Neighbor Fields (NNFs). Unlike standard optical flow, NNFs identify optimal patch-level matches across frames, capturing the scene's motion more effectively for stylization purposes.
- Image Remapping: The stylized frames are warped or remapped based on the NNFs to align with a target frame. This ensures that stylistic elements applied to an object in one frame are consistently positioned in subsequent frames.
- Video Blending: Aligned frames within a sliding window are blended together. This averaging process effectively cancels out the stochastic noise and flickering introduced by the diffusion model.
Technical Innovations
To ensure both efficiency and visual fidelity, the authors introduced two key technical components:
- Remapping Table: To circumvent the $O(NM)$ complexity of naive blending, a tree-like data structure was implemented. This enables a "Fast Mode" that reduces computational complexity to $O(N \log N)$.
- Accurate Mode: A specialized alignment loss function minimizes variance between remapped pixels, preventing the blurriness or "smoggy" artifacts that can occur during simple frame averaging.
Experimental Results
The framework was evaluated using the Pixabay100 dataset, a collection of 100 diverse videos with custom stylization prompts. FastBlend demonstrated superior performance across both quantitative and qualitative metrics:
- Temporal Consistency: FastBlend achieved a Pixel-MSE of 35.79, a significant improvement over the previous state-of-the-art, All-In-One Deflicker (53.38).
- Efficiency: On an NVIDIA RTX 4090, FastBlend processed 100 frames in 2.27 minutes, more than twice as fast as All-In-One Deflicker (5.42 minutes).
- User Preference: In double-blind human evaluations, FastBlend was preferred over 75% of the time against other post-processing methods and over 89% of the time against zero-shot diffusion methods.
Conclusion
FastBlend offers a highly efficient, model-agnostic solution to the persistent problem of flickering in AI-generated videos. While it faces challenges with extremely rapid motion—which can lead to texture blurring—it represents a major step forward in practical video stylization. Future work will focus on integrating more advanced motion estimation to handle high-speed action and expanding the framework's utility to general video restoration tasks.
Summary updated