Abstract
This paper introduces Content-Based 3D Mosaics (CB3M), a novel representation for efficiently storing and analyzing massive video streams from moving cameras. By integrating multi-view pushbroom mosaicing with segmentation-based stereo matching, the CB3M framework transforms raw pixel data into an object-oriented 3D map. This approach achieves compression ratios exceeding 10,000:1 while simultaneously enabling 3D scene reconstruction and the detection of moving targets.
Introduction
The proliferation of mobile imaging platforms, such as drones and aerial vehicles, has led to an explosion of video data. Archiving and searching this data is challenging because traditional video compression (e.g., MPEG) treats video as a sequence of 2D frames, ignoring the underlying 3D structure and the significant motion parallax inherent in translating camera paths. The proposed CB3M framework addresses these inefficiencies by creating a content-aware 3D representation that distinguishes between static terrain and dynamic objects.
Methodology
The CB3M generation process involves two main stages:
1. Multi-View Pushbroom Mosaicing
Instead of standard perspective frames, the system generates "pushbroom" mosaics. This is analogous to a scanner moving across a document, where the camera "scans" the earth from a moving platform. Using a ray interpolation technique (PRISM), the system creates multiple mosaics from different viewing angles (leading, center, and trailing). This geometry, shown in Figure 1, allows the system to capture the scene from multiple perspectives using a single moving camera.
2. Segmentation-Based Stereo Matching
The reference mosaic is segmented into homogeneous color patches, which are assumed to be planar surfaces in 3D space. The system then performs a three-step matching process:
- Global Matching: To find initial displacement.
- Local Matching: Using interest points on patch boundaries as "natural matching primitives."
- Surface Fitting: A RANSAC-based approach to determine the 3D plane parameters for each patch.
Moving targets are identified by detecting "epipolar violations"—patches that do not conform to the static 3D geometry of the scene based on the camera's motion.
Experimental Results
The CB3M framework was evaluated on both simulated and real-world datasets.
Simulated Scene
In a 1,640-frame sequence of a virtual city, the system achieved a compression ratio of 76,061:1, reducing 1.4 GB of raw video to a 19.4 KB file. The 3D reconstruction was highly accurate, with an absolute average error of 0.312 meters and 88.8% of points within ±4 meters of the ground truth.
Real Aerial Video
The system was tested on a 1,000-frame sequence captured from an airplane at 300 meters altitude. It achieved a compression ratio of approximately 10,001:1. As shown in Figure 2, the system successfully generated a high-quality mosaic, a corresponding depth map, and a segmented object map identifying moving vehicles with motion vectors.
Conclusion
The CB3M framework demonstrates that motion parallax, often viewed as a complication in mosaicing, can be leveraged to create highly compressed, semantically rich 3D maps. By shifting from a pixel-based to an object-based representation, the system enables efficient long-term archiving and automated scene analysis. Future work will focus on handling non-planar organic shapes and improving occlusion management through multi-view integration.
Summary updated