Abstract
Video inpainting—the process of filling missing regions in a video—is a challenging task due to complex object motion and the requirement for temporal consistency. Traditional approaches often rely on explicit optical flow estimation to align reference frames, but these methods frequently fail in the very regions that need restoration. This paper introduces the Temporal Adaptive Alignment Network (TAAN), which employs an implicit alignment strategy using deformable convolutions. By aligning features rather than pixels, TAAN produces smoother, more realistic video restoration without the artifacts common to flow-based methods.
The Challenge of Temporal Consistency
The primary goal of video inpainting is to ensure that restored regions do not flicker or warp across frames. To fill a "hole" in a target frame, models typically leverage information from neighboring reference frames.
Standard techniques utilize optical flow to calculate pixel-wise motion. However, estimating accurate flow is notoriously difficult within missing regions. Inaccuracies lead to "ghosting" or blurry artifacts. While some methods use global transformations, these are insufficient for scenes with independent object motion or dynamic camera perspectives.
Proposed Architecture: TAAN
The authors propose an end-to-end network that avoids explicit motion estimation. The architecture is composed of four primary modules:
- Encoder: Extracts high-level features from the target frame and a set of reference frames.
- Temporal Adaptive Alignment (TAA) Module: This core component replaces optical flow with deformable convolutions. It learns offsets for convolution kernels, allowing the network to adaptively sample features from reference frames. This implicit alignment is more robust to noise and missing data than pixel-level warping.
- Feature Aggregation: This module uses a softmax-weighted approach to prioritize information from the most relevant reference frames based on global similarity to the target.
- Reconstruction: A series of dilated convolutions synthesizes the final pixels, followed by up-sampling to the original resolution.
To maintain long-term consistency, the model updates the input sequence with its own previously completed frames, creating a recurrent-like memory of restored content.
Key Insights
- Implicit vs. Explicit Alignment: By using deformable convolutions, the model performs "implicit" motion estimation. It identifies useful features in a neighborhood rather than requiring a perfect vector field, making it significantly more forgiving.
- Feature-Level Synergy: Aligning in the feature space allows the model to leverage semantic context, guiding the inpainting process more effectively than raw pixel manipulation.
- Robustness to Large Holes: The TAA module employs a U-Net-style architecture to estimate offsets even when large portions of the frame are missing, bridging gaps that typically break flow-based algorithms.
Experimental Results
The researchers evaluated TAAN on the YouTube-VOS and DAVIS benchmarks.
Quantitative Performance
TAAN demonstrated clear superiority over existing deep learning baselines such as VINet and CPNet. On the DAVIS dataset, TAAN achieved a PSNR of 30.829, a significant improvement over CPNet's 29.121.
Qualitative Quality
Visual comparisons highlight TAAN's ability to maintain structural integrity. In challenging sequences, such as a dog running through a series of poles, TAAN preserved textures and boundaries that appeared "melted" or warped in flow-based results.
Efficiency
While traditional patch-based methods can produce high-quality results, they are often too slow for practical use. TAAN offers comparable or superior quality while being significantly faster, making it more viable for real-world applications.
Conclusion and Future Work
TAAN demonstrates that implicit feature alignment is a powerful alternative to explicit motion estimation for video inpainting. The authors suggest that this "adaptive alignment" logic is a general-purpose tool that could be extended to other video restoration tasks, such as super-resolution or deblurring.
A remaining challenge is the reliance on temporal windows; if an object is occluded throughout the entire clip, the model must rely on generative synthesis rather than alignment. Future work may focus on integrating stronger generative priors to handle such long-term occlusions.
Summary updated