Thought-deciphering via hierarchical EEG-to-text alignment

Xiaopeng Mao, Morten Mørup, Xin Xu, Sang Wan Lee, Sadasivan Puthusserypady

TL;DR

Researchers have developed a novel hierarchical framework that deciphers human thoughts from EEG signals by first identifying the broad "topic" of a thought before pinpointing the specific sentence. This "divide-and-conquer" approach significantly outperforms direct decoding, achieving up to 87.4% top-10 retrieval accuracy for sentences read from a novel.

What Problem Does This Paper Address?

Brain-Computer Interfaces (BCIs) that translate neural activity into text could restore communication for individuals with severe motor disabilities like locked-in syndrome. While high-end neuroimaging (like fMRI or MEG) has shown impressive results, these systems are bulky and expensive. Electroencephalography (EEG) is portable and affordable, but its low signal-to-noise ratio makes decoding "thought-to-text" notoriously difficult.

Most existing EEG studies focus on tiny units like phonemes or individual words. However, in real-world communication, sentences are the primary currency of meaning. The challenge is that the search space for sentences is massive. This paper addresses the "modality gap" between noisy EEG signals and complex linguistic structures by leveraging the semantic context of what a person is thinking about.

Methods

The authors propose a two-stage hierarchical alignment framework. Instead of trying to find one sentence out of thousands, the model works in steps:

  1. Stage 1: Semantic Classification: The system first aligns EEG representations with broad semantic classes (like a book chapter or a conversation topic). It uses a supervised contrastive loss to ensure that EEG signals from the same topic cluster together in a shared embedding space with the text.
  2. Stage 2: Sentence Retrieval: Once the topic is identified, the search space narrows. The model then performs a fine-grained alignment to retrieve the specific sentence within that predicted topic.
Fig. 1: Flow diagram of the study. Two different raw reading EEG datasets are used. A simple and fast preprocessing strategy is applied to both datasets (a). An EEG encoder converts an EEG sample into a 256-dimensional embedding, which is contrastively aligned with the corresponding text embedding at a class level. The encoded embeddings are used for semantic classification. Subsequently, a retrieval model is used to align EEG embeddings with text embeddings at a sentence level (b). For evaluation, the semantic classifier is evaluated using macro F1 scores, while the retriever limits its text search within the predicted class to find the target sentence (c).

Technically, the architecture uses a frozen BERT model (further compressed via an autoencoder) as a "teacher" for the text embeddings. The EEG encoder consists of EEGNet (for spatial features) followed by a Transformer (for temporal features). The final classification is performed using a k-Nearest Neighbors (KNN) algorithm in the learned embedding space.

Datasets

The study validated the approach on two public datasets:

  • ChineseEEG: 10 subjects reading two novels: The Little Prince (short) and Garnett Dream (long). This provides a coherent narrative context.
  • Chisco: 3 subjects reading 39 categories of daily phrases. Unlike the novels, these phrases are independent and lack a continuous story thread.

Key Insights and Ideas

  • Context is a Force Multiplier: The brain responds much more strongly to sentences within a coherent story than to isolated daily phrases. The "gaps" between matched and unmatched EEG-text pairs were significantly wider for the novel datasets.
  • Hierarchical Search Reduces Noise: By narrowing the search from thousands of sentences to just those within a specific chapter, the model avoids "distractor" sentences that might look similar to the target but belong to a different context.
  • The "Aha" Moment on Retrieval: Direct, unstaged alignment only managed a 50.3% top-10 accuracy. Adding the hierarchical stage boosted this to 83.5%. This suggests that the brain's "macro" state (the topic) is much easier to read via EEG than the "micro" state (the specific words).

Key Results

The hierarchical model showed a dramatic performance leap over baseline methods:

  • The Little Prince: Achieved an average top-10 retrieval accuracy of 83.5% (and 87.4% for the best-performing subject). For comparison, a direct (non-hierarchical) alignment only reached 68.3%.
  • Garnett Dream: While the classification of chapters was nearly perfect (100%), the sentence retrieval was lower (42.6% top-10) because the novel is much longer, making the "fine-grained" step harder.
  • Chisco: Performance was significantly lower (5.9% top-10), highlighting that without a strong narrative context, EEG-to-text decoding remains an uphill battle.
Table 1: Results on ChineseEEG and Chisco. The numbers are means (%) ± standard deviations (%). For ChineseEEG, the book chapters are used as classes. For Chisco, the semantic classes from its publication were reused. Firstly, all the EEG samples are clas- sified by the semantic classifier. Secondly, the corresponding text samples are retrieved within their predicted class by the retriever.

Limitations and Future Work

The authors note that decoding effectiveness depends heavily on the contextual richness of the text. The poor performance on the Chisco dataset suggests that "daily phrase" BCIs might need more than just EEG-text alignment to be viable.

Additionally, the current study relies on a "retrieval" task (choosing from a list) rather than "generation" (creating new text). While retrieval is more robust, it limits the user to a predefined vocabulary. The authors suggest that future "EEG-to-text foundation models" should be trained on coherent stories rather than isolated phrases to better capture how the human brain processes language in context.

Summary updated

Abstract

Communication is important for our well-being, but it can be impaired by disabilities, such as locked-in syndrome (LIS), cerebral palsy (CP), and dysarthria. Non-invasive brain-to-text brain-computer interfaces (BCIs) offer a promising solution by providing an alternative communication pathway for people with communication disabilities. However, most recent breakthroughs have relied on large, cumbersome, and costly neuroimaging modalities, such as functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG). Numerous studies have attempted decoding using the lighter and cheaper electroencephalography (EEG), but the results remain inferior compared to the other modalities. More importantly, we do not understand how these modalities are related to text at the letter, word, sentence, and higher semantic levels. Many prior studies have focused on decoding below the sentence level because it is considerably easier to design an experiment around phonemes, letters, and words. However, during communication, sentences play a much more important role than words and letters. This is because sentences carry information at such a high level that they can be understood even if some words and letters are missing. However, within sentence-level decoding, sentence classification is not as well investigated as sentence generation, making it difficult to decide whether a sentence should be retrieved or generated in the context of EEG-to-text decoding. This paper strives to investigate sentence-level EEG-to-text classification via a novel hierarchical alignment framework, where the main goal is to retrieve a read sentence using its context and EEG as inputs.
BibTeX
@misc{mao2026thought,
  author = {Xiaopeng Mao and Morten Mørup and Xin Xu and Sang Wan Lee and Sadasivan Puthusserypady},
  title = {{Thought-deciphering via hierarchical EEG-to-text alignment}},
  year = {2026},
  eprint = {2611.03623},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.03623},
}
Figure