TL;DR
This paper introduces a novel method for simultaneously detecting and removing dynamic objects from a set of multi-view images. Unlike previous approaches that require user input or rely on video sequences, this algorithm works with static image sets and automatically fills occluded regions with static scene content, producing artifact-free results.
What Problem Does This Paper Address?
Capturing photos in dynamic environments, like public spaces, often results in unwanted moving objects (people, vehicles) appearing in images. While single-image inpainting techniques exist to remove these objects, they often require manual masking and can produce unnatural results due to reliance on image statistics or generic models. Existing multi-view methods for object removal typically require 3D reconstruction or user-provided masks. This paper addresses the challenge of automatically detecting and seamlessly removing dynamic objects from a collection of multi-view images without user intervention, filling the removed areas with plausible static scene content.
Methods
The core of the proposed method is a novel algorithm that iteratively scans a reference image and, for each pixel, decides if it belongs to a static or dynamic object. This decision is made by leveraging information from a set of source images.
Here's a breakdown of the key steps:
- Dense Correspondences: To handle potential deformations in multi-view images, the algorithm extracts both CIE Lab mean features and SIFT features for image patches. It then computes dense correspondence maps between the reference image and each source image using optical flow techniques (specifically mentioning "Full flow" for large displacements). Crucially, the algorithm is designed to be robust to potential mismatches on dynamic objects.
- Similarity Map: A similarity map is computed for each source image to quantify the quality of correspondences. This map considers both appearance (CIE Lab and SIFT features) and geometric consistency (using fundamental matrices).
- Dynamic Map and Decision: A dynamic map (L) is maintained, where 0 indicates a dynamic pixel and 1 indicates a static pixel. The algorithm scans the reference image in a row-major order (alternating directions: top-left to bottom-right, then bottom-right to top-left). At each pixel, candidate locations from source images are identified. A clustering algorithm (DBSCAN, chosen because the number of clusters is unknown) is applied to these candidate locations based on their similarity values. If a pixel is classified as dynamic (L(x,y) = 0), its value is updated.
- Removal and Update: When a pixel is identified as dynamic, its value in the reference image is replaced with the corresponding pixel value from a static region in the source images. This replacement is done by selecting the best matching patch from the source images that corresponds to the occluded static content. The algorithm then updates the dense correspondence maps and similarity maps to reflect these changes, ensuring consistency for subsequent steps. This iterative process continues until no dynamic pixels are detected.
Datasets
The paper acknowledges that a dedicated dataset for multi-view dynamic object removal is not publicly available. Therefore, they constructed their own dataset by:
- Sampling frames from the DAVIS dataset [34] at intervals of 6-10 frames to create sets of 5-7 images.
- Extracting multi-view sets from the Freiburg Berkeley Motion Segmentation dataset [5].
- Utilizing existing datasets for dynamic object detection: skateboard, basketball, climbing, playground, and toyball datasets from Dafni et al. [10], and the tennis dataset from [4].
They also mention using the VLFeat implementation for dense SIFT feature descriptors.
Key Insights and Ideas
- Simultaneous Detection and Removal: The core innovation is performing detection and removal in a single, iterative pass, rather than as separate steps. This allows for immediate feedback and refinement.
- Robustness to Dynamic Objects: The method is designed to be insensitive to the quality of correspondences on dynamic objects themselves. This is a significant advantage, as dynamic objects are inherently harder to match consistently.
- Exploiting Static Region Coherency: The algorithm heavily relies on the assumption that static regions exhibit strong coherency across multiple views. This allows for reliable transfer of static content to fill occluded areas.
- No 3D Reconstruction Needed: The approach avoids computationally expensive 3D reconstruction of the scene, making it more efficient and applicable to scenarios where 3D information is not readily available or reliable.
- Artifact-Free Results: The iterative update of correspondence maps and the careful selection of static patches aim to produce visually pleasing results without noticeable artifacts.
Key Results
- Qualitative Results: The paper presents numerous figures (Fig. 1-6) demonstrating the algorithm's performance on various datasets. These figures visually show the dynamic objects being detected (highlighted) and then replaced by static scene content, often revealing previously occluded details like text or symbols. The results appear visually convincing and artifact-free in many cases.
- Quantitative Results: Table 1 compares the dynamic object detection performance (using Jaccard index) against prior work (Dafni et al. [10] and Kanojia et al. [21]) on the CrowdCam image sets. The proposed method achieves comparable or better results, even though its primary focus is simultaneous detection and removal. The authors state that a direct comparison with learning-based single-image inpainting methods is not entirely fair due to the different input requirements.
Limitations and Future Work
- Dependence on Static Region Quality: The quality of detection and removal is stated to depend on the quality of correspondences obtained in the static regions. If static regions are also poorly represented or highly occluded across views, performance might degrade.
- Scan Direction and Iterations: The algorithm requires multiple scans of the reference image to achieve artifact-free results. The number of scans needed can vary.
- Future Work: The authors suggest exploring more sophisticated methods for handling dynamic objects and potentially extending the framework to video sequences. They also mention the possibility of incorporating more advanced image completion techniques.
Summary updated