Moving Object Completion on the Compressed Domain

Yiwei Jiang, De Xu, Na Liu, Congyan LangIEICE Trans. Information and Systems 2009 · OpenReview

Abstract

Moving object completion is a process of completing moving object's missing information based on local structures. Over the past few years, a number of computable algorithms of video completion have been developed, however most of these algorithms are based on the pixel domain. Little theoretical and computational work in video completion is based on the compressed domain. In this paper, a moving object completion method on the compressed domain is proposed. It is composed of three steps: motion field transferring, thin plate spline interpolation and combination. Missing space-time blocks will be completed by placing new motion vectors on them so that the resulting video sequence will have as much global visual coherence with the video portions outside the hole. The experimental results are presented to demonstrate the efficiency and accuracy of the proposed algorithm.

Introduction

Video completion, or inpainting, involves automatically filling in missing or occluded regions of a video sequence using available spatial and temporal data. This is essential for tasks like removing unwanted objects or repairing damaged film. Traditionally, these operations occur in the pixel domain, requiring full video decoding, frame-by-frame processing, and re-encoding. The primary bottleneck is the Inverse Discrete Cosine Transform (IDCT), which is computationally expensive. This paper proposes a method to perform moving object completion directly within the compressed domain, specifically targeting motion vectors in P-frames to achieve significant speed improvements.

Methodology

The proposed algorithm estimates missing motion information through a three-stage process:

  1. Motion Field Transferring: The algorithm identifies "source patches" in other parts of the video that exhibit similar motion characteristics to the area surrounding the target "hole." These motion fields are then transferred into the missing region.
  2. Thin Plate Spline (TPS) Interpolation: To maintain local spatial consistency, TPS interpolation is used to estimate the motion of missing blocks based on the known motion of adjacent blocks.
  3. Kalman Filter Integration: The estimates from the global search (transferring) and local estimation (TPS) are merged using a Kalman filter. This approach treats the two methods as independent sensors, minimizing prediction error to produce a refined, coherent motion field.

To mitigate the inherent noise in compressed-domain motion vectors, a Gaussian template is applied to smooth the motion field prior to processing.

Figure 3: Example of a video sequence with a defined 'hole' or mask (black region) representing missing data to be completed.

Experimental Results

The algorithm was tested on several standard video sequences, including "Lawn," "Beach," and "Ship." In the "Beach" sequence, the method successfully reconstructed the limbs of individuals walking behind an umbrella, demonstrating the algorithm's ability to handle complex occlusions.

Figure 4: Result of the moving object completion on the Beach sequence, showing restored visual coherence.

The compressed domain approach demonstrated superior efficiency compared to pixel-domain baselines. For the "Lawn" sequence, processing time dropped from 25 seconds to 19 seconds. Similarly, the "Phoenixty" clip saw a reduction from 54 seconds to 37 seconds. Qualitative assessments indicate that the visual quality of the completion is comparable to traditional pixel-domain methods.

Conclusion

By operating directly on motion vectors, the proposed method bypasses the costly decoding process required for pixel-domain video editing. While the approach is currently limited by the block-based nature of the DCT and depends on accurate initial object segmentation, it offers a robust and efficient framework for real-time video completion. Future work will focus on improving sub-block precision and integrating compressed-domain segmentation.

Summary updated

Abstract

Moving object completion is a process of completing moving object's missing information based on local structures. Over the past few years, a number of computable algorithms of video completion have been developed, however most of these algorithms are based on the pixel domain. Little theoretical and computational work in video completion is based on the compressed domain. In this paper, a moving object completion method on the compressed domain is proposed. It is composed of three steps: motion field transferring, thin plate spline interpolation and combination. Missing space-time blocks will be completed by placing new motion vectors on them so that the resulting video sequence will have as much global visual coherence with the video portions outside the hole. The experimental results are presented to demonstrate the efficiency and accuracy of the proposed algorithm.
BibTeX
@article{jiang2009moving,
  author = {Yiwei Jiang and De Xu and Na Liu and Congyan Lang},
  title = {{Moving Object Completion on the Compressed Domain}},
  journal = {IEICE Trans. Information and Systems},
  year = {2009},
  url = {https://openreview.net/forum?id=CaUafaknRS},
}
Figure