TL;DR
This paper introduces SISF-CDN, a novel platform for storing, visualizing, and analyzing massive 3D microscopy datasets on High-Performance Computing (HPC) systems. It tackles the bottleneck of data access and management by providing a scalable storage format and a web-based analysis tool, enabling efficient exploration of petabyte-scale biological data.
What Problem Does This Paper Address?
Modern high-throughput microscopy techniques, like light-sheet microscopy, generate enormous amounts of 3D image data, often reaching terabyte to petabyte scales. While institutional HPC clusters offer the necessary computational power, existing software tools struggle to efficiently interface with these large datasets for interactive visualization, annotation, and analysis. This gap limits the ability of researchers to fully leverage the potential of their imaging experiments. Specifically, managing and accessing many small files, a common issue with some existing formats, can overwhelm supercomputing storage quotas. Furthermore, interactive analysis and annotation of these massive datasets often require specialized, locally installed software, hindering collaborative efforts and accessibility.
Methods
The core of the solution is the Scalable Image Storage Format (SISF). SISF organizes data into hierarchical "shards" (individual files) based on voxel resolution and spatial location, with a two-level indexing system. This approach alleviates the "many small files" problem seen in formats like OME-Zarr. A key innovation is the separation of shard metadata from global image metadata, allowing for flexible storage strategies (e.g., fast SSDs for metadata, cheaper HDDs/tape for data). SISF also incorporates a "virtual stitching" feature that handles overlapping regions from tiled microscopy experiments without requiring data trimming.
To enable efficient access to SISF data, the authors developed the SISF-CDN (Content Delivery Network) server. This server application manages petabyte-scale datasets and provides an API compatible with the Google Neuroglancer precomputed image format. This allows for reslicing data on the fly and downloading only requested subsets, with the computational workload handled by the server.
For annotation and analysis, they built nTracer2, a web-based application that leverages the SISF-CDN. nTracer2 allows multiple users to collaboratively annotate neuron morphology directly in a web browser, with the semi-automated tracing computation handled by the CDN, keeping the analysis within the HPC environment. Neuron tracing data is stored in a custom SQLite3 schema for efficient real-time editing and parsing.
Datasets
The paper utilizes two primary datasets for demonstration and benchmarking:
- ChAT-IRES-Cre mouse brain data: Used for benchmarking SISF and Zarr performance (Fig. 1b,c, Supp. Fig. 5, 6). This dataset was processed using the miriEx16 expansion microscopy protocol and imaged with a Zeiss LSM 780 confocal microscope.
- Thy1-GFP line M mouse brain slice: A large, 295TB dataset (37 teravoxels per channel) used to demonstrate the full-stack capabilities of SISF-CDN and nTracer2 (Fig. 2). This dataset was prepared using a modified miriEx16 expansion microscopy protocol and imaged with a custom line confocal microscope.
Additionally, a previously published green fluorescent protein (GFP) fluorescence micro-optical sectioning tomography (fMOST) brain dataset was used to validate SISF-CDN performance (Fig. 5).
Key Insights and Ideas
- Decoupling Metadata and Data Storage: Separating shard metadata from global image metadata in SISF is a clever way to optimize storage costs and performance by allowing different storage tiers (SSD vs. HDD/tape) to be used effectively.
- Virtual Stitching for Seamless Tiling: The "virtual stitching" feature elegantly handles overlapping regions in tiled images, simplifying data acquisition and visualization without losing information or requiring complex pre-processing.
- CDN for Scalable Web-Based Access: Leveraging a CDN architecture for data access is crucial for enabling interactive visualization and analysis of petabyte-scale datasets from any web browser, democratizing access to large biological data.
- Cloud-Based Annotation: Performing neuron tracing and annotation directly in the cloud via nTracer2, powered by the SISF-CDN, streamlines collaborative workflows and eliminates the need for users to download massive datasets locally for analysis.
Key Results
- SISF Performance: SISF's Python read performance is comparable to Zarr on SSDs and exceeds Zarr on HDDs. SISF's Python write performance is superior to Zarr when creating new archives on both SSD and HDD. Real-world write speeds of ~1TB/minute were achieved using the C++ library for ingesting TIFF images to SISF on an SSD storage array.
- SISF-CDN Download Speeds: The SISF-CDN demonstrated consistent download speeds for resliced orthogonal views over the network, even for large datasets.
- nTracer2 Collaboration: The nTracer2 platform successfully enabled multiple users to simultaneously annotate the same image in a collaborative manner.
- Large Dataset Handling: The platform successfully handled a 295TB, 37 teravoxel GFP mouse brain slice, demonstrating its capability for petabyte-scale data.
Limitations and Future Work
- Current Limitations: The paper mentions that the SISF file format design is subject to a US patent application. While not explicitly stated as a limitation, the complexity of implementing a new file format and associated libraries can be a barrier to adoption.
- Future Work: The authors suggest that the SISF-CDN architecture is versatile and can be integrated into various computational environments, including individual lab servers and large-scale institutional HPC systems. They also highlight the potential for further development and integration of their platform into broader bioinformatic analysis pipelines. The paper also mentions that microscopy data is being uploaded to the Brain Image Library (BIL) for public access, indicating a direction towards broader data sharing and accessibility.
Summary updated