Mumpy: Multilateral Temporal-view Pyramid Transformer for Video Inpainting Detection

Ying Zhang, Yuezun Li, Bo Peng, Jiaran Zhou, Huiyu Zhou, Junyu DongBMVC 2024

Abstract

Video inpainting—the process of removing or replacing objects in video—has become increasingly sophisticated, making manual detection nearly impossible for humans. This paper introduces MumPy (Multilateral Temporal-view Pyramid Transformer), a novel architecture designed to detect these forgeries by flexibly combining spatial and temporal clues across different time scales. By utilizing a multilateral encoder and a multi-pyramid decoder incorporating frequency analysis, MumPy significantly outperforms existing state-of-the-art methods on a new, large-scale benchmark dataset, YTVI.

Introduction

As deep generative models advance, video inpainting has become a powerful tool for both creative editing and malicious digital forensics, such as removing evidence from surveillance footage. Current detection methods often rely on fixed spatial or temporal operations, which struggle to adapt to the varying nature of forgeries across different scenes. For instance, high-motion videos require temporal analysis, while static scenes depend more on spatial texture. MumPy addresses these limitations by moving away from "one-size-fits-all" detection.

Proposed Method: MumPy

The MumPy architecture is built on three primary innovations designed to capture the subtle artifacts left by modern inpainting techniques.

The MumPy architecture features a multilateral encoder for multi-scale temporal views and a multi-pyramid decoder that fuses frequency and semantic features.

1. Multilateral Temporal-view Encoder

Unlike standard encoders that process a fixed number of frames, MumPy uses multiple parallel branches. Each branch employs a different "tubelet" length (3D video patches of varying durations), allowing the model to simultaneously capture short-term spatial details and long-term temporal flow.

2. Deformable Window-based Temporal-view Interaction (DWTI)

To facilitate communication between these branches, the authors developed the DWTI module. This specialized attention mechanism uses deformable attention to "zoom in" on suspicious regions and boundaries where inpainting artifacts are most likely to occur, rather than processing the entire frame uniformly.

3. Multi-pyramid Decoder

The final detection map is generated by aggregating three distinct feature sources:

  • Temporal-view features from the multilateral encoder.
  • Frequency signals (via Discrete Cosine Transform) to identify high-frequency noise discrepancies invisible to the human eye.
  • High-level semantic guidance to maintain context regarding the objects being tracked.

Datasets and Evaluation

The model was evaluated on standard benchmarks (DVI and FVI) and a new, more challenging dataset: YouTube-VOS Video Inpainting (YTVI). YTVI contains 3,471 videos with nearly 6,000 object instances, incorporating seven different state-of-the-art inpainting methods like ProPainter and E2FGVI. This diversity ensures the benchmark reflects modern forgery threats.

Experimental Results

MumPy consistently established a new state-of-the-art (SOTA) across all evaluated metrics.

Qualitative comparison of MumPy against other detection methods (VIDNet, FAST, OSNet, HiFi-Net) across different video sequences.

On the YTVI dataset, MumPy achieved an mIoU of 0.72 and an F1 score of 0.82, outperforming the previous best model (OSNet) by approximately 10.5% and 11.5%, respectively. Furthermore, the model demonstrated strong cross-domain generalization, maintaining a 6% lead when trained on DVI and tested on FVI. Robustness tests also showed that MumPy remains effective under common degradations like JPEG compression and Gaussian noise.

Conclusion

MumPy represents a significant step forward in video digital forensics. By adaptively prioritizing spatial, temporal, and frequency clues, it provides a flexible and robust solution for detecting sophisticated video inpainting. Future work will focus on optimizing the architecture for real-time detection and improving pixel-level accuracy under extreme video compression.

Summary updated

BibTeX
@inproceedings{zhang2024mumpy,
  author = {Ying Zhang and Yuezun Li and Bo Peng and Jiaran Zhou and Huiyu Zhou and Junyu Dong},
  title = {{Mumpy: Multilateral Temporal-view Pyramid Transformer for Video Inpainting Detection}},
  booktitle = {BMVC},
  year = {2024},
}
Figure