EEGUnity: Open-Source Tool in Facilitating Unified EEG Datasets Toward Large-Scale EEG Model

Chengxuan Qin, Rui Yang, Wenlong You, Zhige Chen, Longsheng Zhu, Mengjie Huang, Zidong Wang

Abstract

While large-scale foundation models have revolutionized natural language processing and computer vision, the field of electroencephalography (EEG) has lagged behind due to extreme data heterogeneity. This paper introduces EEGUnity, an open-source Python-based tool designed to streamline the curation of massive EEG datasets. By decoupling metadata from raw signals and leveraging Large Language Models (LLMs) for automated parsing, EEGUnity provides a unified pipeline for data cleaning, standardization, and pretraining readiness. We demonstrate the tool's efficacy across 25 diverse datasets comprising over 40,000 hours of recording time.

Introduction

The success of modern artificial intelligence is largely predicated on the availability of massive, standardized datasets. In the domain of brain-computer interfaces and clinical neurology, researchers aspire to build "Foundation Models" for brain activity. However, unlike the relatively uniform nature of text or image data, EEG data is characterized by a "wild west" of formats and standards.

Existing analytical tools, such as MNE-Python or EEGLAB, are optimized for single-study analysis but lack the infrastructure required to manage dozens of disparate datasets simultaneously. EEGUnity addresses this gap by providing a management layer that transforms heterogeneous raw files into a standardized, machine-learning-ready pipeline.

The Challenge of Heterogeneity

EEG research faces a significant roadblock in data integration due to three primary factors:

  1. Format Proliferation: Data is stored in a variety of formats including .edf, .mat, .csv, and .gdf, each requiring specific parsing logic.
  2. Inconsistent Metadata: Labeling conventions vary wildly between laboratories; for instance, a specific electrode might be labeled "C3" in one study and "Channel_1" in another.
  3. Hardware Variations: Differences in sampling rates, sensor sensitivities, and electrode montages make direct cross-dataset comparisons nearly impossible without extensive manual intervention.

The EEGUnity Framework

EEGUnity is structured around a modular architecture designed to automate the most labor-intensive aspects of data engineering.

The Locator System

At the heart of the tool is the Locator, a metadata management system structured as a Pandas DataFrame. The Locator stores critical information—file paths, sampling rates, channel configurations, and quality scores—separately from the raw data. This design allows researchers to perform non-destructive data correction; one can edit the Locator in a standard spreadsheet to fix errors without altering the original source files, ensuring high reproducibility.

LLM-Boosted Parsing

To handle non-standard file structures that traditional readers fail to recognize, EEGUnity introduces "Large Language Model Boosted" inference. By leveraging models like GPT-4, the tool can interpret descriptive text files or unconventional headers to "guess" channel names and sampling rates. This application of AI to solve "boring" data cleaning problems significantly reduces the manual workload for researchers.

Correction and Batch Processing

The framework includes a dedicated Correction Module for visual inspection and error correction across thousands of files. Furthermore, it provides standardized batch processing pipelines for:

  • Resampling: Aligning all data to a common frequency.
  • Montage Alignment: Mapping diverse electrode configurations to a standard system.
  • Noise Removal: Automated Independent Component Analysis (ICA) for artifact rejection.

Experimental Validation

To evaluate the versatility of EEGUnity, the authors processed 25 distinct EEG datasets sourced from repositories such as Zenodo, PhysioNet, and the Temple University Hospital (TUH) EEG Corpus. This collection represents a massive scale of data:

  • Volume: Over 40,000 hours of recording time.
  • Diversity: Data covering sleep stages, epilepsy, motor imagery, and mental arithmetic.

The tool successfully unified these datasets, generating automated diagnostic reports including magnitude-frequency curves and channel correlation maps. In a test workflow, the authors utilized EEGUnity’s quality scoring to filter data by completeness, retaining approximately 64% of the total data for high-quality model training.

Discussion and Future Work

EEGUnity represents a significant step toward "pretraining readiness" for EEG foundation models. By automating the ingestion and unification of diverse signals, it allows researchers to focus on model architecture rather than data cleaning.

However, technical tools alone cannot solve all hurdles. The authors identify two primary remaining challenges: the legal complexities of sharing sensitive medical data across jurisdictions and the ongoing lack of a universal community standard for EEG metadata. Future development of EEGUnity will focus on supporting niche data formats and facilitating the creation of a massive, publicly accessible unified dataset to jumpstart the next generation of large-scale EEG AI.

Summary updated

Abstract

The increasing number of dispersed EEG dataset publications and the advancement of large-scale Electroencephalogram (EEG) models have increased the demand for practical tools to manage diverse EEG datasets. However, the inherent complexity of EEG data, characterized by variability in content data, metadata, and data formats, poses challenges for integrating multiple datasets and conducting large-scale EEG model research. To tackle the challenges, this paper introduces EEGUnity, an open-source tool that incorporates modules of "EEG Parser", "Correction", "Batch Processing", and "Large Language Model Boost". Leveraging the functionality of such modules, EEGUnity facilitates the efficient management of multiple EEG datasets, such as intelligent data structure inference, data cleaning, and data unification. In addition, the capabilities of EEGUnity ensure high data quality and consistency, providing a reliable foundation for large-scale EEG data research. EEGUnity is evaluated across 25 EEG datasets from different sources, offering several typical batch processing workflows. The results demonstrate the high performance and flexibility of EEGUnity in parsing and data processing. The project code is publicly available at github.com/Baizhige/EEGUnity.
BibTeX
@article{qin2025eegunity,
  author = {Chengxuan Qin and Rui Yang and Wenlong You and Zhige Chen and Longsheng Zhu and Mengjie Huang and Zidong Wang},
  title = {{EEGUnity: Open-Source Tool in Facilitating Unified EEG Datasets Toward Large-Scale EEG Model}},
  journal = {IEEE Transactions on Neural Systems and Rehabilitation Engineering},
  year = {2025},
  doi = {10.1109/tnsre.2025.3565158},
}
Figure