Abstract
Video inpainting evaluation has traditionally lacked the granularity needed to understand specific model failure modes. This paper introduces DEVIL, a diagnostic benchmark designed to move beyond aggregate scoring. By utilizing a specialized landscape dataset and "attribute-aware" diagnostic slicing, DEVIL isolates factors such as camera motion, background complexity, and mask dynamics. This approach reveals critical trade-offs between a model's ability to "hallucinate" new content versus "transferring" existing pixels, providing a more rigorous stress test for state-of-the-art inpainting methods.
Introduction
Video inpainting—the process of filling in missing pixels in a video sequence—is significantly more challenging than its image-based counterpart. Beyond spatial plausibility, models must maintain temporal consistency to prevent flickering and motion artifacts.
The authors argue that current evaluation methods are insufficient. Standard benchmarks often rely on aggregate reconstruction scores from datasets like DAVIS or YouTube-VOS, which are biased toward foreground objects. These methods fail to explain why a model fails. For example, a model might perform well when camera motion reveals occluded background information but fail completely in static shots where it must invent textures from scratch. DEVIL addresses this by providing a framework to pinpoint how specific video attributes impact inpainting quality.
Methods
The DEVIL benchmark introduces a two-part diagnostic solution:
- The DEVIL Dataset: A collection of 1,250 high-quality, "foreground-free" landscape clips. This allows researchers to focus exclusively on how models handle complex background textures (e.g., flowing water, swaying trees) and various motion types.
- Diagnostic Slicing: Instead of a single score, the benchmark evaluates models on "slices" of data where one attribute is held fixed (e.g., "High Camera Motion") while others vary. This allows for a targeted analysis of specific model capabilities.
Performance is measured across three axes:
- Reconstruction: Measured via LPIPS and a new video-based variant, PVCS.
- Realism: Evaluated using FID and Video FID (V-FID).
- Temporal Consistency: Assessed through Pixel-wise Consistency (PCons).
Datasets and Procedural Masking
The benchmark utilizes 1,250 landscape clips sourced from Flickr, annotated for Camera Motion (cm) and Background Scene Motion (bgsm). To simulate missing regions, the authors developed a system for generating procedural masks with controlled dynamics:
- Displacement (fgd): The movement of the mask across the frame.
- Pose Motion (fgpm): Changes in the mask's shape over time.
- Size (fgs): The total area covered by the mask.
Key Insights and Ideas
- The Importance of Flow: Top-performing models (JointOpt, DFCNet, FGVC) all explicitly calculate optical flow during inference. This "task-driven flow" is essential for maintaining sharp, motion-consistent results.
- Hallucination vs. Transfer: High camera motion can actually improve performance for some models like VINet. This is because camera movement "reveals" the background over time, allowing the model to transfer real pixels rather than having to hallucinate them.
- Consistency vs. Quality: High temporal consistency scores can be misleading; some models achieve them by producing blurry, static results that lack realism. Consistency must be balanced against reconstruction and realism.
Key Results
The authors benchmarked seven state-of-the-art (SOTA) methods, leading to several key findings:
- Top Performers: JointOpt, DFCNet, and FGVC consistently outranked others by treating time as an ordered structure with long-range dependencies.
- Mid-Tier Struggles: Transformer-based (STTN) and Copy-Paste (CPNet) methods struggle when camera motion is high, as their internal alignment mechanisms fail to map information across distant frames correctly.
- No Universal Winner: No single method wins across every attribute slice. A model might excel at handling moving water but fail during rapid camera pans, highlighting the need for diagnostic evaluation.
Limitations and Future Work
While DEVIL provides deep insights into background inpainting, it currently lacks the complexity of interacting foreground objects (e.g., a person walking behind a tree). Future iterations could benefit from a continuous scale of difficulty rather than binary "Low/High" attributes. The authors suggest that future research should focus on improving "hallucination" in static-camera scenarios and developing better ways to propagate information across long temporal gaps without losing sharpness.
Summary updated