Gigapixel behavioral and neural activity imaging with a novel multi-camera array microscope

Eric E Thomson, Mark Harfouche, Pavan Chandra Konda, Catherine W Seitz, Kanghyun Kim, Colin Cooke, Shiqi Xu, Whitney S Jacobs, Robin Blazing, Yang Chen, Sunanda Sharma, Timothy Dunn, Jaehee Park, Roarke Horstmeyer, Eva A. NaumannbioRxiv 2021 · DOI

TL;DR

Researchers have developed a novel "gigapixel" microscope (MCAM) using an array of up to 96 cameras to simultaneously capture high-resolution images of large areas, enabling the study of complex behaviors and neural activity in freely moving organisms across multiple spatial scales. This system overcomes previous limitations in field-of-view and resolution, offering a powerful new tool for biological research.

What Problem Does This Paper Address?

Observing biological systems often requires bridging vast differences in spatial scales. For example, social behaviors like schooling in fish occur across centimeters, while the underlying neural activity happens at the micrometer scale. Current imaging systems typically force a trade-off: either a large field-of-view (FOV) with low resolution, or high resolution over a very small area. This makes it difficult to simultaneously study both large-scale behaviors and fine-grained cellular processes in freely moving organisms. Existing methods often require tethering animals, which restricts natural behavior, or use complex single-lens systems that become prohibitively large, expensive, and suffer from optical aberrations. The challenge is to achieve both gigapixel-scale FOV and high spatial resolution without these limitations.

Methods

The core innovation is the Multi-Camera Array Microscope (MCAM). This system uses an array of inexpensive, readily available CMOS image sensors (similar to those in smartphones), each equipped with its own lens. These "micro-cameras" are arranged in a grid with overlapping fields of view.

  • Scalable Architecture: The MCAM is modular, allowing for scaling up to 96 cameras (arranged in a 2x2 grid of 24-camera units) to achieve a gigapixel-scale image.
  • Parallel Acquisition: All cameras capture data simultaneously, avoiding the temporal ambiguities of sequential scanning methods.
  • Image Stitching: Custom software is used to stitch the individual camera images into a single, high-resolution composite image.
  • Stereoscopic Depth Tracking: The overlapping fields of view between adjacent cameras are used to calculate the z-position (depth) of organisms, enabling 3D tracking.
  • Dual-Channel Fluorescence Imaging: The system was adapted for fluorescence imaging, allowing for simultaneous recording of different fluorescent signals (e.g., red and green) for ratiometric measurements of neural activity.
  • Machine Learning for Analysis: Advanced machine learning techniques, including CNN-based object detection and siamese neural networks for individual identification, were adapted to handle the massive datasets generated.

Datasets

The paper utilizes a variety of model organisms for demonstration and validation, rather than specific training datasets in the traditional machine learning sense. These include:

  • Larval zebrafish (wild-type and transgenic strains)
  • Fruit flies (Drosophila melanogaster)
  • Nematodes (Caenorhabditis elegans - wild-type and mutant strains)
  • Carpenter ants (Camponotus pennsylvanicus)
  • Slime mold (Physarum polycephalum)

The datasets generated are described as "gigapixel videos" of these organisms performing various behaviors or exhibiting neural activity.

Key Insights and Ideas

  • Bridging Scales is Crucial: The paper strongly emphasizes that understanding complex biological systems requires observing phenomena across vastly different spatial scales simultaneously.
  • Modularity is Key to Scalability: The MCAM's modular design, using many inexpensive sensors, is a practical and cost-effective way to achieve gigapixel resolution, overcoming the limitations of complex single-lens systems.
  • Overlapping FOV Enables More Than Just Stitching: The overlap between camera views is not just for seamless image assembly but is critical for enabling advanced functionalities like stereoscopic depth tracking and ratiometric fluorescence measurements.
  • AI is Essential for Gigapixel Data: The sheer volume of data generated necessitates sophisticated machine learning algorithms for detection, tracking, and analysis, pushing the boundaries of what's possible with large-scale imaging.
  • Uncovering Novel Behaviors: The ability to observe large groups of organisms in their natural state, at high resolution, can reveal previously unobserved collective behaviors (e.g., swarming in C. elegans).

Key Results

  • Gigapixel Resolution: The system can achieve gigapixel-scale images (up to 0.96 gigapixels per frame) with a field of view of hundreds of square centimeters.
  • High Spatial Resolution: The MCAM demonstrates an 18 µm full-pitch optical resolution and single-point sensitivity down to 3-5 µm.
  • 3D Tracking: Stereoscopic depth tracking provides approximately 100 µm accuracy in depth localization over a >5 mm range.
  • Neural Activity Monitoring: The system successfully recorded regionally localized neural activity in freely moving zebrafish using ratiometric calcium imaging, correlating activity with locomotion.
  • Behavioral Observation: The MCAM was used to image and analyze the behavior of diverse organisms, including swarming in C. elegans, maze-solving in slime mold, and light-response in ants.
  • Data Throughput: A continuous recording session at 0.74 Hz generated 2.5 TB of unstitched data and 1.17 TB of stitched data in under an hour.

Limitations and Future Work

  • Frame Rate: The current full-frame acquisition speed of 1 Hz is noted as being too slow for some applications requiring faster kinematic analysis. Future work aims to increase frame rates through improved data transfer (PCIe), on-chip processing (binning, cropping), and dynamic pre-processing on FPGAs.
  • Data Handling: While novel display technologies can visualize gigapixel videos, the sheer size of the raw data (terabytes) remains a challenge. Future work will focus on improving processing software for automated analysis and extracting salient information.
  • Single-Neuron Resolution: While the MCAM can measure regionally localized brain activity, it currently cannot detect single-neuron fluorescence activity in the zebrafish brain.
  • Commercialization: The authors note that two of the authors are co-founders of Ramona Optics Inc., which is commercializing and patenting the multi-camera array microscope, indicating a path towards wider adoption.

Summary updated

Abstract

Abstract The dynamics of living organisms are organized across many spatial scales. However, current cost-effective imaging systems can measure only a subset of these scales at once. We have created a scalable multi-camera array microscope (MCAM) that enables comprehensive high-resolution recording from multiple spatial scales simultaneously, ranging from cellular structures to large-group behavioral dynamics. By collecting data from up to 96 cameras, we computationally generate gigapixel-scale images and movies with a field of view over hundreds of square centimeters at 5um sensitivity. This allows us to observe the behavior and fine anatomical features of numerous freely moving model organisms on multiple spatial scales, including larval zebrafish, fruit flies, nematodes, carpenter ants, and slime mold. The MCAM architecture allows stereoscopic tracking of the z-position of organisms using the overlapping field of view from adjacent cameras. Further, we demonstrate the ability to acquire dual color fluorescence video of multiple freely moving zebrafish, recording neural activity via ratiometric calcium imaging. Overall, the MCAM provides a powerful platform for investigating cellular and behavioral processes across a wide range of spatial scales by removing the bottlenecks imposed by single-camera image acquisition systems.
BibTeX
@misc{thomson2021gigapixel,
  author = {Eric E Thomson and Mark Harfouche and Pavan Chandra Konda and Catherine W Seitz and Kanghyun Kim and Colin Cooke and Shiqi Xu and Whitney S Jacobs and Robin Blazing and Yang Chen and Sunanda Sharma and Timothy Dunn and Jaehee Park and Roarke Horstmeyer and Eva A. Naumann},
  title = {{Gigapixel behavioral and neural activity imaging with a novel multi-camera array microscope}},
  year = {2021},
  doi = {10.1101/2021.10.05.461695},
  howpublished = {bioRxiv},
}

Loading related content...

Figure