A modular hierarchical array camera

Xiaoyun Yuan, Mengqi Ji, Jiamin Wu, David J. Brady, Qionghai Dai, Lu Fang

Abstract

Array cameras removed the optical limitations of a single camera and paved the way for high-performance imaging via the combination of micro-cameras and computation to fuse multiple aperture images. However, existing solutions use dense arrays of cameras that require laborious calibration and lack flexibility and practicality. Inspired by the cognition function principle of the human brain, we develop an unstructured array camera system that adopts a hierarchical modular design with multiscale hybrid cameras composing different modules. Intelligent computations are designed to collaboratively operate along both intra- and intermodule pathways. This system can adaptively allocate imagery resources to dramatically reduce the hardware cost and possesses unprecedented flexibility, robustness, and versatility. Large scenes of real-world data were acquired to perform human-centric studies for the assessment of human behaviours at the individual level and crowd behaviours at the population level requiring high-resolution long-term monitoring of dynamic wide-area scenes.

Introduction

Traditional high-performance imaging often relies on array cameras: clusters of micro-cameras that fuse multiple images into a single gigapixel-scale frame. However, existing designs typically follow a "uniform sensation principle," assuming information is evenly distributed across a scene. This leads to massive, bulky systems that waste computational and hardware resources capturing empty backgrounds at the same resolution as dynamic targets. Furthermore, these structured arrays require laborious calibration and lack the flexibility to adapt to different environments.

To address this "data deluge" and hardware inefficiency, we present UnstructuredCam, a system that mimics how the human brain prioritizes information through hierarchical processing and modular organization.

System Architecture and Methods

The core of the system is a hierarchical modular design consisting of two distinct layers:

  1. Global Layer: A single wide-angle camera that captures the entire field of view (FoV) to provide context and coordination.
  2. Local Layer: Multiple high-resolution micro-cameras mounted on ball gimbals. These cameras are "unstructured," meaning they can be pointed independently at specific regions.
The hierarchical design of UnstructuredCam is inspired by the rich-club organization of the human brain, where global hubs coordinate specialized local modules.

To decide where the local cameras should look, the system calculates a temporal entropy map from the global view. This map identifies regions with high dynamic activity (such as moving vehicles or pedestrians). An optimization algorithm then assigns the local cameras to maximize the "covered information" based on this entropy.

Furthermore, an unstructured embedding algorithm allows for online recalibration, making the system robust to physical disturbances. For larger scenes, multiple modules can collaborate (inter-module collaboration) to generate 3D gigapixel panoramas using a novel trinocular depth estimation algorithm.

Key Insights: Information Sparsity and Robustness

  • Rich-Club Organization: Just as the brain uses "hubs" to integrate information from various specialized regions, UnstructuredCam uses the global camera as a hub to coordinate specialized local sensors.
  • Information Sparsity: Natural scenes are inherently sparse. By focusing pixels only where "action" happens (high temporal entropy), the system drastically reduces hardware costs without losing meaningful data.
  • Robustness through Hierarchy: Because the local cameras are independent and coordinated by a global "anchor," the system is far more resilient to individual sensor failure or movement than rigid, pre-calibrated arrays.

Experimental Results

The system was evaluated using the PANDA dataset (Gigapixel-level Human-Centric Video), which contains real-world video sequences of large-scale outdoor scenes.

  • Efficiency Gain: The study found that 20 cameras using this unstructured strategy provided the same information coverage as 65 cameras in a conventional structured array, representing a 70% reduction in hardware costs.
  • Real-time Performance: The system supports real-time gigapixel streaming and storage on standard PC platforms, overcoming previous data processing bottlenecks.
  • Large-Scale Analysis: The system successfully tracked over 4,000 people in a marathon race, capturing facial details at distances up to 100 meters while simultaneously modeling overall crowd density and flow.
Demonstration of the system's ability to capture high-resolution details (e.g., license plates and faces) at distances ranging from 400 to 700 meters within a wide-area scene.

Conclusion

UnstructuredCam represents a shift from "uniform sensation" to "content-adaptive sampling." While the system introduces mechanical complexity via gimbals and currently prioritizes dynamic over static objects, its efficiency and flexibility pave the way for practical wide-area monitoring. Future work will explore integrating neuromorphic computing directly into the camera units to further decentralize processing and mimic neuronal behavior.

Summary updated

Abstract

Array cameras removed the optical limitations of a single camera and paved the way for high-performance imaging via the combination of micro-cameras and computation to fuse multiple aperture images. However, existing solutions use dense arrays of cameras that require laborious calibration and lack flexibility and practicality. Inspired by the cognition function principle of the human brain, we develop an unstructured array camera system that adopts a hierarchical modular design with multiscale hybrid cameras composing different modules. Intelligent computations are designed to collaboratively operate along both intra- and intermodule pathways. This system can adaptively allocate imagery resources to dramatically reduce the hardware cost and possesses unprecedented flexibility, robustness, and versatility. Large scenes of real-world data were acquired to perform human-centric studies for the assessment of human behaviours at the individual level and crowd behaviours at the population level requiring high-resolution long-term monitoring of dynamic wide-area scenes.
BibTeX
@inproceedings{yuan2021modular,
  author = {Xiaoyun Yuan and Mengqi Ji and Jiamin Wu and David J. Brady and Qionghai Dai and Lu Fang},
  title = {{A modular hierarchical array camera}},
  booktitle = {Light},
  year = {2021},
  doi = {10.1038/s41377-021-00485-x},
}
Figure