Abstract
Video inpainting—the process of removing or replacing objects in video—has become increasingly sophisticated, making manual detection nearly impossible for humans. This paper introduces MumPy (Multilateral Temporal-view Pyramid Transformer), a novel architecture designed to detect these forgeries by flexibly combining spatial and temporal clues across different time scales. By utilizing a multilateral encoder and a multi-pyramid decoder incorporating frequency analysis, MumPy significantly outperforms existing state-of-the-art methods on a new, large-scale benchmark dataset, YTVI.
Introduction
As deep generative models advance, video inpainting has become a powerful tool for both creative editing and malicious digital forensics, such as removing evidence from surveillance footage. Current detection methods often rely on fixed spatial or temporal operations, which struggle to adapt to the varying nature of forgeries across different scenes. For instance, high-motion videos require temporal analysis, while static scenes depend more on spatial texture. MumPy addresses these limitations by moving away from "one-size-fits-all" detection.
Proposed Method: MumPy
The MumPy architecture is built on three primary innovations designed to capture the subtle artifacts left by modern inpainting techniques.
1. Multilateral Temporal-view Encoder
Unlike standard encoders that process a fixed number of frames, MumPy uses multiple parallel branches. Each branch employs a different "tubelet" length (3D video patches of varying durations), allowing the model to simultaneously capture short-term spatial details and long-term temporal flow.
2. Deformable Window-based Temporal-view Interaction (DWTI)
To facilitate communication between these branches, the authors developed the DWTI module. This specialized attention mechanism uses deformable attention to "zoom in" on suspicious regions and boundaries where inpainting artifacts are most likely to occur, rather than processing the entire frame uniformly.
3. Multi-pyramid Decoder
The final detection map is generated by aggregating three distinct feature sources:
- Temporal-view features from the multilateral encoder.
- Frequency signals (via Discrete Cosine Transform) to identify high-frequency noise discrepancies invisible to the human eye.
- High-level semantic guidance to maintain context regarding the objects being tracked.
Datasets and Evaluation
The model was evaluated on standard benchmarks (DVI and FVI) and a new, more challenging dataset: YouTube-VOS Video Inpainting (YTVI). YTVI contains 3,471 videos with nearly 6,000 object instances, incorporating seven different state-of-the-art inpainting methods like ProPainter and E2FGVI. This diversity ensures the benchmark reflects modern forgery threats.
Experimental Results
MumPy consistently established a new state-of-the-art (SOTA) across all evaluated metrics.
On the YTVI dataset, MumPy achieved an mIoU of 0.72 and an F1 score of 0.82, outperforming the previous best model (OSNet) by approximately 10.5% and 11.5%, respectively. Furthermore, the model demonstrated strong cross-domain generalization, maintaining a 6% lead when trained on DVI and tested on FVI. Robustness tests also showed that MumPy remains effective under common degradations like JPEG compression and Gaussian noise.
Conclusion
MumPy represents a significant step forward in video digital forensics. By adaptively prioritizing spatial, temporal, and frequency clues, it provides a flexible and robust solution for detecting sophisticated video inpainting. Future work will focus on optimizing the architecture for real-time detection and improving pixel-level accuracy under extreme video compression.
Summary updated