Gigapixel imaging with a novel multi-camera array microscope

Eric E Thomson, Mark Harfouche, Kanghyun Kim, Pavan Chandra Konda, Catherine W Seitz, Colin Cooke, Shiqi Xu, Whitney S Jacobs, Robin Blazing, Yang Chen, Sunanda Sharma, Timothy Dunn, Jaehee Park, Roarke Horstmeyer, Eva A. NaumanneLife 2022 · DOI

TL;DR

This paper introduces the Multi-Camera Array Microscope (MCAM), a novel, scalable, and cost-effective system capable of capturing gigapixel-resolution videos of freely moving organisms across large fields of view. This breakthrough overcomes a fundamental trade-off in microscopy, enabling simultaneous observation of both macroscopic behaviors and fine anatomical details, opening new avenues for studying complex biological systems.

What Problem Does This Paper Address?

Observing biological systems often requires capturing phenomena that span vastly different spatial scales, from the collective behavior of entire populations to the subtle movements of individual cells or anatomical features. Traditional microscopes face a critical trade-off: high magnification and resolution come at the cost of a very narrow field of view, while wide fields of view necessitate sacrificing resolution. This limitation hinders the study of complex behaviors and interactions in freely moving organisms, where both large-scale dynamics and fine-grained details are crucial for understanding. Existing solutions, like head-fixing animals or using time-sequential scanning, often restrict natural behavior or introduce temporal ambiguities.

Methods

The core innovation is the Multi-Camera Array Microscope (MCAM). This system is built by tiling numerous inexpensive CMOS image sensors, each equipped with its own lens, into a large array. The key features include:

  • Scalable Architecture: The system is modular, allowing for arrays of 24 (MCAM-24) or 96 (MCAM-96) cameras, enabling a flexible field of view.
  • Parallel Acquisition: Up to 96 cameras capture data simultaneously, generating gigapixel-scale images and videos.
  • Image Stitching: Custom software and commercial tools (PtGui) are used to stitch together the individual camera feeds into a single, coherent image. This process relies on calibration targets to extract precise alignment parameters.
  • Stereoscopic Depth Tracking: By strategically overlapping the fields of view of adjacent cameras, the system can perform stereoscopic depth estimation, allowing for 3D tracking of organisms.
  • Deep Learning for Analysis:
    • Object Detection: A Faster R-CNN model, pre-trained on COCO and fine-tuned on MCAM data, is used to detect and localize individual organisms within the gigapixel images. This required novel strategies for handling the massive image sizes, including sub-image processing and custom augmentation for occlusion.
  • Zebrafish Identification: A Siamese neural network with triplet loss was developed to distinguish individual larval zebrafish based on subtle anatomical differences (melanophore patterns), a feat not previously demonstrated at this developmental stage.
  • C. elegans Segmentation: A U-Net architecture was employed to segment individual worms and swarms for quantitative analysis of collective behavior.

Datasets

The paper utilizes datasets generated from experiments with various model organisms:

  • Zebrafish: Larval zebrafish (5-10 days post-fertilization) were used for object detection, identification, and 3D tracking.
  • C. elegans: Multiple strains (wild-type, transgenic, and mutant) were used to study swarming behavior.
  • Slime Mold (Physarum polycephalum): Used to demonstrate macroscopic maze traversal and cytoplasmic flow.
  • Carpenter Ants (Camponotus spp.): Studied for behavioral responses to light cycles.
  • Fruit Flies (Drosophila melanogaster): Used to observe social interactions and developmental stages.

The authors also mention using standard datasets for pre-training object detection models (e.g., COCO).

Key Insights and Ideas

  • Breaking the FOV-Resolution Barrier: The MCAM architecture fundamentally overcomes the traditional trade-off by parallelizing image acquisition across many sensors, enabling both large fields of view and high resolution simultaneously.
  • Scalability is Key: The modular design allows the system to be scaled to meet different experimental needs, from smaller arrays to gigapixel-scale imaging.
  • Deep Learning for Gigapixel Data: The paper demonstrates that advanced deep learning techniques can be effectively adapted to process and extract meaningful information from massive gigapixel datasets, even with limited GPU memory.
  • Individual Identification in Larvae: The ability to distinguish individual larval zebrafish based on subtle anatomical features is a significant advancement, enabling more precise behavioral studies at this developmental stage.
  • 3D Tracking in Large Groups: The stereoscopic depth tracking capability allows for the reconstruction of 3D trajectories of multiple organisms within a large arena, providing a richer understanding of collective dynamics.

Key Results

  • Gigapixel Imaging: The MCAM system successfully generates gigapixel-scale images (up to 0.96 gigapixels) and videos.
  • Resolution: Achieved an optical resolution of 18 µm full-pitch.
  • Field of View: Demonstrated imaging over hundreds of square centimeters (e.g., 16x24 cm).
  • Frame Rate: Achieved up to 1 Hz in full-frame mode, with potential for higher rates through optimizations.
  • Stereoscopic Depth Accuracy: Demonstrated an approximate resolution of 100 µm in depth localization over a >5 mm depth range.
  • Zebrafish Identification: A Siamese network successfully distinguished individual larval zebrafish with high accuracy.
  • C. elegans Swarming: Enabled quantitative analysis of novel swarming behaviors in different C. elegans strains.
  • Cost-Effectiveness: The system's cost scales linearly with the number of pixels, making it significantly less expensive than comparable large-area, high-resolution imaging systems.

Limitations and Future Work

  • Frame Rate: The current 1 Hz frame rate in full-frame mode is insufficient for tracking very fast movements. Future work aims to increase frame rates through hardware and software optimizations (e.g., faster data transfer, on-chip processing, updated USB protocols).
  • Fluorescence Imaging: While demonstrated, fluorescence imaging is limited by factors like axial blurring, potential photobleaching, and the use of Bayer-patterned sensors which reduce sensitivity and resolution per channel. Future work will focus on improving sensitivity, exploring strobed excitation, and potentially using unfiltered monochrome pixels for higher resolution.
  • Cellular Resolution: The current 18 µm resolution is not sufficient for resolving individual cells unless sparsely labeled. Tighter camera packing and improved optics are suggested for achieving cellular-scale detail.
  • Data Management: Handling terabytes of raw data requires efficient processing pipelines, which the authors are actively developing.

The authors envision the MCAM as a versatile platform for a wide range of applications, including industrial inspection, pathology specimen analysis, and monitoring of bacterial cultures, in addition to its primary use in biological research.

Summary updated

Abstract

The dynamics of living organisms are organized across many spatial scales. However, current cost-effective imaging systems can measure only a subset of these scales at once. We have created a scalable multi-camera array microscope (MCAM) that enables comprehensive high-resolution recording from multiple spatial scales simultaneously, ranging from structures that approach the cellular scale to large-group behavioral dynamics. By collecting data from up to 96 cameras, we computationally generate gigapixel-scale images and movies with a field of view over hundreds of square centimeters at an optical resolution of 18 µm. This allows us to observe the behavior and fine anatomical features of numerous freely moving model organisms on multiple spatial scales, including larval zebrafish, fruit flies, nematodes, carpenter ants, and slime mold. Further, the MCAM architecture allows stereoscopic tracking of the z-position of organisms using the overlapping field of view from adjacent cameras. Overall, by removing the bottlenecks imposed by single-camera image acquisition systems, the MCAM provides a powerful platform for investigating detailed biological features and behavioral processes of small model organisms across a wide range of spatial scales.
BibTeX
@inproceedings{thomson2022gigapixel,
  author = {Eric E Thomson and Mark Harfouche and Kanghyun Kim and Pavan Chandra Konda and Catherine W Seitz and Colin Cooke and Shiqi Xu and Whitney S Jacobs and Robin Blazing and Yang Chen and Sunanda Sharma and Timothy Dunn and Jaehee Park and Roarke Horstmeyer and Eva A. Naumann},
  title = {{Gigapixel imaging with a novel multi-camera array microscope}},
  booktitle = {eLife},
  year = {2022},
  doi = {10.7554/elife.74988},
}
Figure