ROAMM: A Benchmark Dataset for Multimodal Human Attention Decoding and EEG-to-Text Modeling During Naturalistic Reading

Haorui Sun, Ardyn Vivienne Olszko, Niharika Singh, David C. JangrawICML 2026 · OpenReview

Introduction: The Challenge of Natural Reading

Humans spend between 30% and 60% of their waking lives mind-wandering (MW)—thinking about a past argument or a future task while their eyes continue to scan a page. For researchers building Brain-Computer Interfaces (BCIs) designed to translate neural signals into text, this natural cognitive fluctuation presents a significant hurdle.

Most existing datasets for "EEG-to-text" decoding rely on tightly controlled, sentence-by-sentence reading tasks. While foundational, these setups fail to reflect the messy, multi-page reality of natural reading where attention inevitably drifts. Until now, the field has lacked a large-scale dataset combining high-fidelity brain recordings with precise labels for exactly when a reader’s mind wanders. Without this data, developing "attention-aware" AI that knows when to trust the neural signals it processes remains impossible.

The ROAMM Dataset and Methodology

To bridge this gap, researchers have introduced ROAMM (Reading Observed At Mindless Moments), a massive multimodal dataset featuring 50 hours of simultaneous EEG and eye-tracking data from 44 participants engaged in naturalistic reading. Unlike previous datasets like ZuCo, which focus on isolated sentences, ROAMM captures multi-page reading dynamics and is the first publicly available dataset to provide span-level MW onset and offset labels alongside co-registered physiological data.

The ReMind Paradigm

Capturing the exact moment a mind wanders is notoriously difficult. The authors utilized the ReMind paradigm, a retrospective self-report method. Participants read five long Wikipedia articles at their own pace. Upon realizing their mind had wandered, they pressed a key and retroactively highlighted the specific text span where their focus drifted and subsequently returned.

Technical Setup

  • Physiological Recording: A 64-channel BioSemi EEG system recording at 2048 Hz, paired with an SR Research EyeLink 1000 Plus eye-tracker at 1000 Hz.
  • Decoding Architecture: A pre-trained BART (Large Language Model) fine-tuned with an EEG encoder to translate brain waves back into the read text.
  • Evaluation Protocol: The team employed a strict Leave-One-Subject-Out (LOSO) evaluation to ensure the model learned generalizable features rather than memorizing individual brain patterns.

Key Insights: Perceptual Decoupling and Multimodal Detection

The ROAMM dataset provides empirical support for the "perceptual decoupling" effect. During mind-wandering, the brain partially disconnects from external stimuli. While the eyes may continue to move across the words—outwardly simulating reading—the higher-level neural integration required for language processing fundamentally breaks down.

Detecting these shifts is challenging, but the study demonstrates that a multimodal approach is highly effective. Combining eye-tracking (which captures behavioral shifts like reduced fixation rates) with EEG (which captures neural shifts in theta and alpha bands) consistently outperformed unimodal detection methods.

Results: The Impact of Attention on Decoding

The most striking finding from the ROAMM benchmark is how profoundly attention alters neural language representations.

  • Mind-Wandering Detection: Using a Random Forest classifier on combined EEG and eye-tracking features, the researchers achieved an AUROC of 0.609 under the rigorous LOSO protocol. While seemingly modest, this performance is highly competitive given the unconstrained, naturalistic setting.
  • Decoding Performance Collapse: When the EEG-to-text model was applied to mind-wandering segments, its performance plummeted. The BLEU-1 score (measuring text overlap) dropped from 0.189 during attentive reading to a mere 0.057 during mind-wandering episodes. This confirms that attention is not merely a "volume knob" for brain activity; it fundamentally changes how language is represented in the cortex.

Limitations and Future Directions

While ROAMM represents a major step forward, the authors note several limitations:

  1. Coarse Comprehension Metrics: The study used multiple-choice questions to assess understanding, which may miss the nuances of partial comprehension.
  2. Undifferentiated MW Content: The current labels indicate when a participant's mind wandered, but not where it went (e.g., a blank mind versus a vivid memory). Different types of MW likely produce distinct neural signatures.
  3. Inter-individual Variability: Brain patterns vary significantly across individuals. Despite the success of the LOSO evaluation, there remains a performance gap compared to models trained on a single individual's data.

Ultimately, ROAMM opens the door for the next generation of attention-aware language models—AI systems that do not just process text, but actively understand the cognitive state of the human reading it.

Summary updated

Abstract

We present Reading Observed At Mindless Moments (ROAMM), a multimodal dataset comprising 50 hours of simultaneous EEG and eye-tracking recordings collected during naturalistic multi-page reading from 44 participants. ROAMM includes synchronized physiological recordings, eye-movement events, page-level comprehension scores, and span-level mind-wandering (MW) annotations obtained using a retrospective self-report paradigm. We introduce a standardized leave-one-subject-out benchmark for MW detection and achieve up to 0.609 AUROC using supervised models. We additionally evaluate EEG-to-text decoding on reading segments with and without MW labels, showing that decoding performance decreases during MW episodes. ROAMM enables research on MW detection, EEG-to-text decoding, multimodal representation learning, and attention-related degradation of language representations during naturalistic reading.
BibTeX
@inproceedings{sun2026roamm,
  author = {Haorui Sun and Ardyn Vivienne Olszko and Niharika Singh and David C. Jangraw},
  title = {{ROAMM: A Benchmark Dataset for Multimodal Human Attention Decoding and EEG-to-Text Modeling During Naturalistic Reading}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=zqLPdt09fE},
}
Figure