Abstract
Array cameras removed the optical limitations of a single camera and paved the way for high-performance imaging via the combination of micro-cameras and computation to fuse multiple aperture images. However, existing solutions use dense arrays of cameras that require laborious calibration and lack flexibility and practicality. Inspired by the cognition function principle of the human brain, we develop an unstructured array camera system that adopts a hierarchical modular design with multiscale hybrid cameras composing different modules. Intelligent computations are designed to collaboratively operate along both intra- and intermodule pathways. This system can adaptively allocate imagery resources to dramatically reduce the hardware cost and possesses unprecedented flexibility, robustness, and versatility. Large scenes of real-world data were acquired to perform human-centric studies for the assessment of human behaviours at the individual level and crowd behaviours at the population level requiring high-resolution long-term monitoring of dynamic wide-area scenes.
Introduction
Traditional high-performance imaging often relies on array cameras: clusters of micro-cameras that fuse multiple images into a single gigapixel-scale frame. However, existing designs typically follow a "uniform sensation principle," assuming information is evenly distributed across a scene. This leads to massive, bulky systems that waste computational and hardware resources capturing empty backgrounds at the same resolution as dynamic targets. Furthermore, these structured arrays require laborious calibration and lack the flexibility to adapt to different environments.
To address this "data deluge" and hardware inefficiency, we present UnstructuredCam, a system that mimics how the human brain prioritizes information through hierarchical processing and modular organization.
System Architecture and Methods
The core of the system is a hierarchical modular design consisting of two distinct layers:
- Global Layer: A single wide-angle camera that captures the entire field of view (FoV) to provide context and coordination.
- Local Layer: Multiple high-resolution micro-cameras mounted on ball gimbals. These cameras are "unstructured," meaning they can be pointed independently at specific regions.
To decide where the local cameras should look, the system calculates a temporal entropy map from the global view. This map identifies regions with high dynamic activity (such as moving vehicles or pedestrians). An optimization algorithm then assigns the local cameras to maximize the "covered information" based on this entropy.
Furthermore, an unstructured embedding algorithm allows for online recalibration, making the system robust to physical disturbances. For larger scenes, multiple modules can collaborate (inter-module collaboration) to generate 3D gigapixel panoramas using a novel trinocular depth estimation algorithm.
Key Insights: Information Sparsity and Robustness
- Rich-Club Organization: Just as the brain uses "hubs" to integrate information from various specialized regions, UnstructuredCam uses the global camera as a hub to coordinate specialized local sensors.
- Information Sparsity: Natural scenes are inherently sparse. By focusing pixels only where "action" happens (high temporal entropy), the system drastically reduces hardware costs without losing meaningful data.
- Robustness through Hierarchy: Because the local cameras are independent and coordinated by a global "anchor," the system is far more resilient to individual sensor failure or movement than rigid, pre-calibrated arrays.
Experimental Results
The system was evaluated using the PANDA dataset (Gigapixel-level Human-Centric Video), which contains real-world video sequences of large-scale outdoor scenes.
- Efficiency Gain: The study found that 20 cameras using this unstructured strategy provided the same information coverage as 65 cameras in a conventional structured array, representing a 70% reduction in hardware costs.
- Real-time Performance: The system supports real-time gigapixel streaming and storage on standard PC platforms, overcoming previous data processing bottlenecks.
- Large-Scale Analysis: The system successfully tracked over 4,000 people in a marathon race, capturing facial details at distances up to 100 meters while simultaneously modeling overall crowd density and flow.
Conclusion
UnstructuredCam represents a shift from "uniform sensation" to "content-adaptive sampling." While the system introduces mechanical complexity via gimbals and currently prioritizes dynamic over static objects, its efficiency and flexibility pave the way for practical wide-area monitoring. Future work will explore integrating neuromorphic computing directly into the camera units to further decentralize processing and mimic neuronal behavior.
Summary updated