Temporal Adaptive Alignment Network for Deep Video Inpainting

Ruixin Liu, Zhenyu Weng, Yuesheng Zhu, Bairong Li

Abstract

Video inpainting—the process of filling missing regions in a video—is a challenging task due to complex object motion and the requirement for temporal consistency. Traditional approaches often rely on explicit optical flow estimation to align reference frames, but these methods frequently fail in the very regions that need restoration. This paper introduces the Temporal Adaptive Alignment Network (TAAN), which employs an implicit alignment strategy using deformable convolutions. By aligning features rather than pixels, TAAN produces smoother, more realistic video restoration without the artifacts common to flow-based methods.

The Challenge of Temporal Consistency

The primary goal of video inpainting is to ensure that restored regions do not flicker or warp across frames. To fill a "hole" in a target frame, models typically leverage information from neighboring reference frames.

Standard techniques utilize optical flow to calculate pixel-wise motion. However, estimating accurate flow is notoriously difficult within missing regions. Inaccuracies lead to "ghosting" or blurry artifacts. While some methods use global transformations, these are insufficient for scenes with independent object motion or dynamic camera perspectives.

Proposed Architecture: TAAN

The authors propose an end-to-end network that avoids explicit motion estimation. The architecture is composed of four primary modules:

  1. Encoder: Extracts high-level features from the target frame and a set of reference frames.
  2. Temporal Adaptive Alignment (TAA) Module: This core component replaces optical flow with deformable convolutions. It learns offsets for convolution kernels, allowing the network to adaptively sample features from reference frames. This implicit alignment is more robust to noise and missing data than pixel-level warping.
  3. Feature Aggregation: This module uses a softmax-weighted approach to prioritize information from the most relevant reference frames based on global similarity to the target.
  4. Reconstruction: A series of dilated convolutions synthesizes the final pixels, followed by up-sampling to the original resolution.
The TAAN architecture consists of an encoder, the Temporal Adaptive Alignment (TAA) module, feature aggregation, and a reconstruction network.

To maintain long-term consistency, the model updates the input sequence with its own previously completed frames, creating a recurrent-like memory of restored content.

Key Insights

  • Implicit vs. Explicit Alignment: By using deformable convolutions, the model performs "implicit" motion estimation. It identifies useful features in a neighborhood rather than requiring a perfect vector field, making it significantly more forgiving.
  • Feature-Level Synergy: Aligning in the feature space allows the model to leverage semantic context, guiding the inpainting process more effectively than raw pixel manipulation.
  • Robustness to Large Holes: The TAA module employs a U-Net-style architecture to estimate offsets even when large portions of the frame are missing, bridging gaps that typically break flow-based algorithms.

Experimental Results

The researchers evaluated TAAN on the YouTube-VOS and DAVIS benchmarks.

Quantitative Performance

TAAN demonstrated clear superiority over existing deep learning baselines such as VINet and CPNet. On the DAVIS dataset, TAAN achieved a PSNR of 30.829, a significant improvement over CPNet's 29.121.

Qualitative Quality

Visual comparisons highlight TAAN's ability to maintain structural integrity. In challenging sequences, such as a dog running through a series of poles, TAAN preserved textures and boundaries that appeared "melted" or warped in flow-based results.

Qualitative comparison showing TAAN's superior performance in maintaining object structure and texture compared to other state-of-the-art methods.

Efficiency

While traditional patch-based methods can produce high-quality results, they are often too slow for practical use. TAAN offers comparable or superior quality while being significantly faster, making it more viable for real-world applications.

Conclusion and Future Work

TAAN demonstrates that implicit feature alignment is a powerful alternative to explicit motion estimation for video inpainting. The authors suggest that this "adaptive alignment" logic is a general-purpose tool that could be extended to other video restoration tasks, such as super-resolution or deblurring.

A remaining challenge is the reliance on temporal windows; if an object is occluded throughout the entire clip, the model must rely on generative synthesis rather than alignment. Future work may focus on integrating stronger generative priors to handle such long-term occlusions.

Summary updated

Abstract

Video inpainting aims to synthesize visually pleasant and temporally consistent content in missing regions of video. Due to a variety of motions across different frames, it is highly challenging to utilize effective temporal information to recover videos. Existing deep learning based methods usually estimate optical flow to align frames and thereby exploit useful information between frames. However, these methods tend to generate artifacts once the estimated optical flow is inaccurate. To alleviate above problem, we propose a novel end-to-end Temporal Adaptive Alignment Network(TAAN) for video inpainting. The TAAN aligns reference frames with target frame via implicit motion estimation at a feature level and then reconstruct target frame by taking the aggregated aligned reference frame features as input. In the proposed network, a Temporal Adaptive Alignment (TAA) module based on deformable convolutions is designed to perform temporal alignment in a local, dense and adaptive manner. Both quantitative and qualitative evaluation results show that our method significantly outperforms existing deep learning based methods.
BibTeX
@inproceedings{liu2020temporal,
  author = {Ruixin Liu and Zhenyu Weng and Yuesheng Zhu and Bairong Li},
  title = {{Temporal Adaptive Alignment Network for Deep Video Inpainting}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/129},
}
Figure