Content-Based Dynamic 3D Mosaics

Zhigang Zhu, Hao TangWorkshops @ CVPR 2006 · DOI

Abstract

This paper introduces Content-Based 3D Mosaics (CB3M), a novel representation for efficiently storing and analyzing massive video streams from moving cameras. By integrating multi-view pushbroom mosaicing with segmentation-based stereo matching, the CB3M framework transforms raw pixel data into an object-oriented 3D map. This approach achieves compression ratios exceeding 10,000:1 while simultaneously enabling 3D scene reconstruction and the detection of moving targets.

Introduction

The proliferation of mobile imaging platforms, such as drones and aerial vehicles, has led to an explosion of video data. Archiving and searching this data is challenging because traditional video compression (e.g., MPEG) treats video as a sequence of 2D frames, ignoring the underlying 3D structure and the significant motion parallax inherent in translating camera paths. The proposed CB3M framework addresses these inefficiencies by creating a content-aware 3D representation that distinguishes between static terrain and dynamic objects.

Methodology

The CB3M generation process involves two main stages:

1. Multi-View Pushbroom Mosaicing

Instead of standard perspective frames, the system generates "pushbroom" mosaics. This is analogous to a scanner moving across a document, where the camera "scans" the earth from a moving platform. Using a ray interpolation technique (PRISM), the system creates multiple mosaics from different viewing angles (leading, center, and trailing). This geometry, shown in Figure 1, allows the system to capture the scene from multiple perspectives using a single moving camera.

Geometry of pushbroom mosaicing from a moving aerial platform, illustrating the relationship between camera motion and scene scanning.

2. Segmentation-Based Stereo Matching

The reference mosaic is segmented into homogeneous color patches, which are assumed to be planar surfaces in 3D space. The system then performs a three-step matching process:

  • Global Matching: To find initial displacement.
  • Local Matching: Using interest points on patch boundaries as "natural matching primitives."
  • Surface Fitting: A RANSAC-based approach to determine the 3D plane parameters for each patch.

Moving targets are identified by detecting "epipolar violations"—patches that do not conform to the static 3D geometry of the scene based on the camera's motion.

Experimental Results

The CB3M framework was evaluated on both simulated and real-world datasets.

Simulated Scene

In a 1,640-frame sequence of a virtual city, the system achieved a compression ratio of 76,061:1, reducing 1.4 GB of raw video to a 19.4 KB file. The 3D reconstruction was highly accurate, with an absolute average error of 0.312 meters and 88.8% of points within ±4 meters of the ground truth.

Real Aerial Video

The system was tested on a 1,000-frame sequence captured from an airplane at 300 meters altitude. It achieved a compression ratio of approximately 10,001:1. As shown in Figure 2, the system successfully generated a high-quality mosaic, a corresponding depth map, and a segmented object map identifying moving vehicles with motion vectors.

Reconstruction and analysis of real aerial video: (a) the generated mosaic strip, (b) the estimated depth map, and (c) the segmented scene with detected moving targets and motion vectors.

Conclusion

The CB3M framework demonstrates that motion parallax, often viewed as a complication in mosaicing, can be leveraged to create highly compressed, semantically rich 3D maps. By shifting from a pixel-based to an object-based representation, the system enables efficient long-term archiving and automated scene analysis. Future work will focus on handling non-planar organic shapes and improving occlusion management through multi-view integration.

Summary updated

BibTeX
@inproceedings{zhu2006content,
  author = {Zhigang Zhu and Hao Tang},
  title = {{Content-Based Dynamic 3D Mosaics}},
  booktitle = {Workshops @ CVPR},
  year = {2006},
  doi = {10.1109/cvprw.2006.60},
}
Figure