ZarrFM: A Canonical Tensor Storage Architecture for EEG Foundation-Model Pretraining

Dheeraj Chintapalli, Sunkalp ChandraarXiv 2026 · arXiv

TL;DR

ZarrFM is a new high-performance storage architecture designed specifically for the "execution" phase of EEG foundation model pretraining. By utilizing a Zarr v3 hierarchy with canonical channel mapping and block-aligned sharding, it achieves up to 11x faster data loading than Parquet and reduces metadata overhead by orders of magnitude while cutting storage footprints in half.

What Problem Does This Paper Address?

Training EEG foundation models (like BENDR or LaBraM) requires massive datasets—often tens of thousands of hours of recordings. However, the EEG community lacks a storage format optimized for deep learning.

Existing standards like EDF or BIDS are designed for archiving, not training. This creates a "preprocessing tax": every researcher must independently write code to parse headers, resolve heterogeneous channel names (e.g., mapping "FP1-REF" to a standard "Fp1"), resample signals, and handle subject-level splits. This leads to massive I/O overhead, slow training starts due to metadata "storms" (thousands of small file requests), and results that are difficult to reproduce because every pipeline preprocesses the data slightly differently.

Methods

The authors propose ZarrFM, an execution-oriented representation that materializes a training-ready corpus into a single Zarr v3 store. Key technical innovations include:

  • Canonicalization at Ingestion: Instead of resolving channel identities at runtime, ZarrFM maps signals into a frozen, versioned 112-slot canonical space (91 scalp positions and 21 auxiliary classes) during the initial conversion.
  • Two-Level Consolidated Index: To avoid the latency of opening thousands of individual files, ZarrFM uses a consolidated index. This allows a trainer to initialize a dataset and resolve recording metadata in just a few requests, rather than one per file.
  • Block-Aligned Sharding: Standard random-window sampling is inefficient for compressed data because reading a 4-second window often requires decompressing a much larger chunk. ZarrFM’s WindowSampler (shown in Figure 1) groups windows by their backing storage blocks, reading each block once and emitting all contained windows in a shuffled order.
  • Append-Only Edit Protocol: To maintain data integrity during large-scale maintenance, ZarrFM uses a hash-verified, append-only protocol for annotations and metadata updates, preventing "torn states" or data loss during crashes.
Figure 1: Left: The ZarrFM hierarchy: one corpus and one store containing R recordings and K=112 canonical channel slots, with rk denoting the current event revision. Right: The training path, with subjects and sessions stored as index columns rather than groups.

Datasets

The authors evaluated ZarrFM using:

  • PhysioNet EEG Corpora: A massive conversion of 15 open-access databases (including CHB-MIT, EEGBCI, and Sleep-EDF), totaling 11,877 recordings and 236.5 GB.
  • Synthetic EEG: Used for controlled experiments on variation and anti-aliasing.
  • MOABB (BNCI2014_009): Used to test handling of non-integer (float64) input.

Key Insights and Ideas

  • Archival vs. Execution: The core "aha moment" is that we shouldn't be training directly on archival formats. By moving expensive operations like channel mapping and metadata aggregation to a one-time ingestion step, we can drastically simplify the training loop.
  • The Request Bottleneck: In modern cloud and remote storage (like S3), the number of requests is often a bigger bottleneck than raw bandwidth. ZarrFM’s consolidated index reduces 601 metadata requests down to just 14 for a 200-recording sample.
  • Preserving the Lattice: Unlike many pipelines that cast everything to float32 immediately, ZarrFM preserves the native int16 signal codes. This not only saves space but also ensures bit-exact reproducibility across different programming languages (Python, C++, Rust).

Key Results

ZarrFM outperforms traditional formats across nearly every metric:

  • Throughput: As shown in the table below, ZarrFM with block-merged reads reaches 3,131 windows/s, which is 5x faster than EDF and 11x faster than Parquet.
  • Storage Efficiency: By preserving native signal codes and using efficient compression, ZarrFM reduced the storage of the PhysioNet corpus from 42.4 MB/hour (EDF) to 22.7 MB/hour.
  • Compression: Across the entire 236.5 GB PhysioNet library, ZarrFM achieved a 2.36x compression ratio relative to the original EDF files while remaining bit-exact upon readback.
Table 1: Random-window reads on one CHB-MIT clinical hour (23 ch, 256 Hz): 512 random 4 s windows, all channels, one process, local NVMe; medians of four warm runs. NPY is an uncompressed memmap; Parquet stores pre-cut float32 windows with zstd, following EEG-FM- Bench [39], with 16-window row groups.
Figure 2: All open-access PhysioNet EDF databases: 15 databases, 11,877 recordings, 236.5 GB, 8,359 h, with bit-exact readback. Left: byte-weighted compression relative to EDF; bars show fast, dots max, dashed lines corpus totals. Right: achieved bits/sample versus order-0 entropy. LTRSVP reaches first-difference entropy, while bigP3BCI’s 5× ratio reflects 82 boolean flag channels.

Limitations and Future Work

While ZarrFM offers significant speedups, the authors note several trade-offs:

  • Lossy Canonicalization: On specific clinical montages (like bipolar recordings), distinct channels can "collide" into a single canonical slot, causing data loss.
  • Quantization for Floats: For sources that aren't natively integers, ZarrFM uses a user-defined quantization step, which is not bit-exact to the original float values.
  • Sampler Confounding: The current throughput benchmarks compare ZarrFM's block-aligned sampler against "one-window-at-a-time" baselines; future work should isolate the format's benefits from the sampler's benefits.
  • End-to-End Validation: The paper focuses on I/O and storage; the authors suggest future studies should quantify how these representation choices (like quantization) specifically impact downstream model accuracy.

Summary updated

Abstract

Statistical models for brain-computer interfaces (BCIs) increasingly require large, heterogeneous EEG corpora, yet existing datasets vary in sampling rates, channel layouts, window lengths, and storage formats, creating substantial preprocessing and I/O overhead. We propose ZarrFM, an execution-oriented representation that complements BIDS by storing a corpus as a single Zarr v3 hierarchy with indexed metadata, canonical channel spaces, explicit masks, hash-stable subject splits, verified event tables, and block-aligned sharding. On real PhysioNet recordings and synthetic corpora, we compare ZarrFM with EDF, FIF, NPY, HDF5, and Parquet. Across 200 recordings, its two-level index reduces metadata queries from 601 requests to 14. Preserving native signal codes reduces storage from 42.4 MB/hour in EDF to 22.7 MB/hour, while block-aligned sampling reaches 3,131 windows/s, up to 11 times faster than Parquet. Reads remain bit-identical across independent Zarr implementations, and concurrent failure-injected appends recover without data loss. These results show that ZarrFM can provide a compact, interoperable, and execution-efficient storage layer for large-scale EEG pretraining.
BibTeX
@misc{chintapalli2026zarrfm,
  author = {Dheeraj Chintapalli and Sunkalp Chandra},
  title = {{ZarrFM: A Canonical Tensor Storage Architecture for EEG Foundation-Model Pretraining}},
  year = {2026},
  eprint = {2611.11149},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11149},
}
Figure