TeraVR Empowers Precise Reconstruction of Complete 3-D Neuronal Morphology in the Whole Brain

Yimin Wang, Qi Li, Lijuan Liu, Zhi Zhou, Yun Wang, Lingsheng Kong, Ning Zhong, Renjie Chai, Xiangfeng Luo, Yike Guo, Michael Hawrylycz, Qingming Luo, Zhongze Gu, Wei Xie, Hongkui Zeng, Lili Yuan, Hanchuan PengbioRxiv 2019 · DOI

TL;DR

This paper introduces TeraVR, an open-source virtual reality system designed to tackle the immense challenge of reconstructing the complete 3D morphology of neurons within entire mouse brains. By integrating immersive visualization, collaborative annotation, and efficient teravoxel-scale data streaming, TeraVR significantly improves the accuracy and efficiency of this complex neuroinformatics task.

What Problem Does This Paper Address?

Understanding the brain's structure and function relies heavily on mapping the intricate shapes of its neurons, known as their morphology. However, reconstructing the complete 3D morphology of even a single neuron is incredibly difficult due to complex branching patterns, overlapping structures, and often weak or noisy imaging signals. When this task is scaled up to an entire mouse brain, which can involve tens of teravoxels of data, the challenge becomes exponentially greater. Existing methods struggle with visualizing and interacting with such massive datasets, making accurate and efficient reconstruction a major bottleneck in neuroscience research.

Methods

The core of this paper is the introduction of TeraVR, a virtual reality (VR) annotation system. TeraVR builds upon the TeraFly module of Vaa3D, leveraging its capabilities for handling teravoxel-scale data. Key features of TeraVR include:

  • Immersive 3D Visualization: Utilizes VR headsets (like the HTC Vive) to provide users with an intuitive, stereoscopic view of the brain data, allowing for better spatial understanding of complex neuronal structures.
  • Hierarchical Streaming: Efficiently streams teravoxel-scale image volumes, enabling real-time interaction and exploration of massive datasets without requiring the entire dataset to be loaded into memory.
  • Collaborative Annotation: Implements a cloud-based server-client infrastructure that allows multiple annotators, even geographically dispersed, to work on the same dataset simultaneously. Annotations are merged in real-time, and each annotator is represented by a unique color and avatar.
  • Mixed Reality Demonstration: A technique was developed to combine real-world footage of the annotator with the virtual environment, providing a dynamic and engaging demonstration of the system's use.
  • AI Integration: TeraVR incorporates a trained U-Net deep learning model to assist in separating neurite signals from background noise. Additionally, a data-filtering model is implemented to detect and alert users to potential reconstruction errors, such as branches with sharp turns.
  • Smooth VR/Non-VR Switching: Allows annotators to seamlessly switch between VR and traditional 2D visualization modes to best suit different areas of the imaging dataset.

Datasets

The paper utilizes whole-brain imaging data from mouse brains. Specific mouse lines used for data preparation include:

  • Tnnt1-IRES2-CreERT2;Ai82;Ai140 (brain ID No. 17302 and 17545)
  • Gnb4-IRES2-CreERT2;Ai139 (No. 236174)
  • Plxnd1-CreER;Ai82;Ai140 (No. 17300)

The imaging data was produced using fMOST (Li, et al, 2010) and converted into the TeraFly-format. The paper does not introduce new benchmark datasets but rather applies TeraVR to existing whole-brain datasets.

Key Insights and Ideas

  • VR is a Game-Changer for Big Data Visualization: The immersive nature of VR, combined with efficient data streaming, fundamentally changes how researchers can interact with and understand massive, complex 3D datasets like whole-brain neuronal morphology. It moves beyond the limitations of 2D screens for truly appreciating intricate structures.
  • Collaboration is Crucial for Accuracy: Reconstructing complex neurons is prone to errors and ambiguities. TeraVR's collaborative mode, inspired by Google Docs, allows multiple experts to pool their insights, leading to more robust and accurate reconstructions in less time than a single annotator.
  • AI Augments Human Annotation: Integrating AI tools like U-Net for denoising and error detection doesn't replace human annotators but significantly enhances their efficiency and accuracy by pre-processing data and flagging potential issues.
  • Bridging the Gap Between Resolution and Scale: TeraVR addresses the long-standing challenge of reconstructing fine neuronal details at a whole-brain scale, a feat difficult to achieve with traditional methods that are either high-resolution but limited in scope (EM) or lower-resolution but cover larger areas (LM).

Key Results

  • Efficiency Gains: In challenging scenarios with dense and weakly imaged neurites, TeraVR reduced tracing time by 50-80% compared to the most efficient non-VR approach (TeraFly). For weak neurite signals, reconstruction time was reduced to about 50 seconds, approximately 10 times faster than non-VR methods.
  • Labor Reduction: TeraVR reduced labor by 60%-80% as measured by alternative metrics like the number of strokes required for tracing.
  • Improved Reconstruction Completeness: For 17 complex neurons, TeraVR extended the overall length of reconstructions by 10-103% compared to Neurolucida, indicating it recovered significantly more of the neuronal structure.
  • Collaborative Speedup: Three geographically remote annotators co-reconstructed a neuron in 20% of the time it would take a single annotator.
  • Accuracy Validation: Reconstructions produced by different combinations of annotators using TeraVR showed consistent topology and minimal length differences (0.77% of total length).

Limitations and Future Work

  • VR Hardware Limitations: Currently, VR headsets are single-user devices, limiting simultaneous direct interaction. The collaborative mode addresses this by allowing remote users to join a shared session.
  • AI Model Refinement: The U-Net model can be iteratively refined based on user feedback, suggesting future work in adapting it to different brain images and improving its performance.
  • Data Management for Extremely Large Datasets: While TeraVR handles teravoxels, organizing data in the Vaa3D-TeraFly format is recommended for optimal performance with very large datasets (>100 billion voxels).
  • Broader Application: The authors expect TeraVR to be applicable to other massive-scale datasets from various microscopy techniques beyond just whole-brain neuronal morphology.
  • Further AI Integration: The paper mentions the potential for integrating more machine learning-based data analysis modules to further enhance automation and crowdsourcing of "gold standard" reconstructions.

Summary updated

Abstract

Abstract Neuron morphology is recognized as a key determinant of cell type, yet the quantitative profiling of a mammalian neuron’s complete three-dimensional (3-D) morphology remains arduous when the neuron has complex arborization and long projection. Whole-brain reconstruction of neuron morphology is even more challenging as it involves processing tens of teravoxels of imaging data. Validating such reconstructions is extremely laborious. We developed TeraVR , an open-source virtual reality annotation system, to address these challenges. TeraVR integrates immersive and collaborative 3-D visualization, interaction, and hierarchical streaming of teravoxel-scale images. Using TeraVR , we produced precise 3-D full morphology of long-projecting neurons in whole mouse brains and developed a collaborative workflow for highly accurate neuronal reconstruction.
BibTeX
@misc{wang2019i,
  author = {Yimin Wang and Qi Li and Lijuan Liu and Zhi Zhou and Yun Wang and Lingsheng Kong and Ning Zhong and Renjie Chai and Xiangfeng Luo and Yike Guo and Michael Hawrylycz and Qingming Luo and Zhongze Gu and Wei Xie and Hongkui Zeng and Lili Yuan and Hongkui Zeng and Hanchuan Peng},
  title = {{<i>TeraVR</i> Empowers Precise Reconstruction of Complete 3-D Neuronal Morphology in the Whole Brain}},
  year = {2019},
  doi = {10.1101/621011},
  howpublished = {bioRxiv},
}
Figure