PatchPerPixMatch for Automated 3d Search of Neuronal Morphologies in Light Microscopy

Lisa Mais, P. B. Hirsch, Claire Managan, Kaiyu Wang, Konrad Rokicki, Robert Svirskas, Barry J. Dickson, Wyatt Korff, Gerald M. Rubin, Gudrun Ihrke, Geoffrey W Meissner, Dagmar KainmuellerbioRxiv 2021 · DOI

Abstract

Studies of individual neurons in the Drosophila nervous system are facilitated by transgenic lines that sparsely and repeatably label respective neurons of interest. Sparsity can be enhanced by means of intersectional approaches like the split-GAL4 system, which labels the positive intersection of the expression patterns of two (denser) GAL4 lines. To this end, two GAL4 lines have to be identified as labelling a neuron of interest. Current approaches to tackling this task include visual inspection, as well as automated search in 2d projection images, of single cell multi-color flip-out (MCFO) acquisitions of GAL4 expression patterns. There is to date no automated method available that performs full 3d search in MCFO imagery of GAL4 lines, nor one that leverages automated reconstructions of the labelled neuron morphologies. To close this gap, we propose PatchPerPixMatch, a fully automated approach for finding a given neuron morphology in MCFO acquisitions of Gen1 GAL4 lines. PatchPerPixMatch performs automated instance segmentation of MCFO acquisitions, and subsequently searches for a target neuron morphology by minimizing an objective that aims at covering the target with a set of well-fitting segmentation fragments. Patch-PerPixMatch is computationally efficient albeit being full 3d, while also highly robust to inaccuracies in the automated neuron instance segmentation. We are releasing PatchPerPixMatch search results for ~30,000 neuron morphologies from the Drosophila hemibrain in ~20,000 MCFO acquisitions of ~3,500 Gen1 GAL4 lines.

Introduction

Understanding the functional architecture of the brain requires linking high-resolution "wiring diagrams" from electron microscopy (EM) to the living brains studied via light microscopy (LM). In Drosophila research, a primary goal is identifying genetic "GAL4 lines" that label specific neurons of interest. This identification allows researchers to manipulate those neurons in vivo. However, this process is currently a significant bottleneck. Researchers often manually inspect thousands of 3D image stacks or rely on 2D projection searches like Color-depth Maximum Intensity Projection (MIP), which can discard vital spatial information. While automated 3D segmentation exists, it frequently suffers from "false splits" (breaking a neuron into pieces) or "false merges" (fusing neighbors). PatchPerPixMatch addresses these challenges by performing a full 3D search designed to be robust against such segmentation errors.

Methods: The Jigsaw Puzzle Approach

The PatchPerPixMatch pipeline operates in two distinct phases. First, it performs Instance Segmentation using PatchPerPix, a deep-learning method that segments neurons in Multi-Color Flip-Out (MCFO) images. MCFO is particularly useful because it labels neighboring neurons in different colors, allowing the algorithm to use both shape and color for differentiation.

Second, the system performs Match Optimization. Rather than searching for a single, perfect segmentation, PatchPerPixMatch treats the search like a jigsaw puzzle. Given a target neuron skeleton (e.g., from an EM reconstruction), it identifies a subset of segmentation fragments that best reconstruct the target.

The PatchPerPixMatch fragment selection process. (a) Shows the target neuron morphology. (b) Demonstrates how the algorithm selects multiple fragments (labeled with score/coverage) to reconstruct the target, overcoming the issue of "broken" segmentations.

The optimization objective seeks to maximize coverage of the target skeleton while ensuring a high-quality morphological fit (using NBLAST-like scores) and maintaining color consistency across fragments. This prevents the system from "frankensteining" a match out of fragments that clearly belong to different neurons.

The optimization objective accounts for over-coverage and under-coverage. By penalizing these deviations, the system ensures that the selected fragments accurately represent the target morphology without including extraneous structures.

To handle the scale of ~600 million potential searches, the authors engineered an efficient solver using upper and lower energy bounds to prune the search space, making the combinatorial optimization computationally feasible.

Datasets and Scale

The study leveraged two massive resources: ~30,000 neuron morphologies from the Drosophila hemibrain (EM) and ~20,000 MCFO acquisitions representing ~3,500 Gen1 GAL4 lines. Validation was performed on a curated set of 10 target neurons with 47 known matching GAL4 lines.

Key Results and Insights

The core innovation of PatchPerPixMatch is its robustness through fragmentation. By allowing multiple fragments to cover one target, the system handles "false split" errors—a common failure in automated microscopy—naturally. Furthermore, using color as a constraint provides a powerful filter against incorrect matches.

The system achieved high retrieval accuracy, with a median rank of 23 for known matching acquisitions and a median "line rank" of 18 out of 3,500. It also demonstrated fine-grained sensitivity, successfully distinguishing between highly similar cell types like pC1d and pC1e. Despite the complexity of 3D search, the pipeline is efficient, averaging 116 minutes per target morphology on a single 3.0 GHz core.

Conclusion

PatchPerPixMatch provides a scalable, automated solution for bridging the gap between EM and LM datasets. While challenges like "false merges" and stochastic labeling remain, the tool represents a significant step forward. The authors are releasing search results for the entire Drosophila hemibrain and plan to integrate these findings into the NeuronBridge web tool, making these insights accessible to the global neuroscience community.

Summary updated

Abstract

Abstract Studies of individual neurons in the Drosophila nervous system are facilitated by transgenic lines that sparsely and repeatably label respective neurons of interest. Sparsity can be enhanced by means of intersectional approaches like the split-GAL4 system, which labels the positive intersection of the expression patterns of two (denser) GAL4 lines. To this end, two GAL4 lines have to be identified as labelling a neuron of interest. Current approaches to tackling this task include visual inspection, as well as automated search in 2d projection images, of single cell multi-color flip-out (MCFO) acquisitions of GAL4 expression patterns. There is to date no automated method available that performs full 3d search in MCFO imagery of GAL4 lines, nor one that leverages automated reconstructions of the labelled neuron morphologies. To close this gap, we propose PatchPerPixMatch, a fully automated approach for finding a given neuron morphology in MCFO acquisitions of Gen1 GAL4 lines. PatchPerPixMatch performs automated instance segmentation of MCFO acquisitions, and subsequently searches for a target neuron morphology by minimizing an objective that aims at covering the target with a set of well-fitting segmentation fragments. Patch-PerPixMatch is computationally efficient albeit being full 3d, while also highly robust to inaccuracies in the automated neuron instance segmentation. We are releasing PatchPerPixMatch search results for ~30,000 neuron morphologies from the Drosophila hemibrain in ~20,000 MCFO acquisitions of ~3,500 Gen1 GAL4 lines. Code https://github.com/Kainmueller-Lab/PatchPerPixMatch Results https://pppm.janelia.org
BibTeX
@misc{mais2021patchperpixmatch,
  author = {Lisa Mais and P. B. Hirsch and Claire Managan and Kaiyu Wang and Konrad Rokicki and Robert Svirskas and Barry J. Dickson and Wyatt Korff and Gerald M. Rubin and Gudrun Ihrke and Geoffrey W Meissner and Dagmar Kainmueller},
  title = {{PatchPerPixMatch for Automated 3d Search of Neuronal Morphologies in Light Microscopy}},
  year = {2021},
  doi = {10.1101/2021.07.23.453511},
  howpublished = {bioRxiv},
}
Figure