Abstract
Traditional search engines often struggle with "vocabulary mismatch," where user queries are too vague or brief to retrieve relevant information. This paper presents Brain-Aug, a novel framework that leverages fMRI brain signals to decode user intent and automatically expand search queries. By mapping neural activity directly into the latent space of a Large Language Model (LLM), Brain-Aug generates semantically rich query continuations. Experimental results across multiple datasets demonstrate that this approach significantly improves search accuracy, particularly for ambiguous queries, by bridging the gap between human thought and information retrieval systems.
Introduction
Search engine effectiveness is fundamentally limited by the quality of user-provided queries. Users frequently submit short, ambiguous, or poorly phrased queries, leading to a "vocabulary mismatch" between the query and the target documents. While traditional Query Augmentation techniques like Pseudo-Relevance Feedback (PRF) attempt to mitigate this by expanding queries using top-ranked search results, they are prone to failure if the initial results are irrelevant—a "garbage in, garbage out" scenario.
This research proposes a paradigm shift: utilizing the user's internal cognitive state to disambiguate and enrich search queries. By decoding brain activity during the search process, the system can capture the user's true intent more accurately than text alone.
Methodology
The Brain-Aug framework integrates neuroimaging data with modern Information Retrieval (IR) techniques through four primary stages.
- Brain Decoder: A Multi-Layer Perceptron (MLP) is employed to translate fMRI (functional Magnetic Resonance Imaging) features into the latent embedding space of a Large Language Model (LLM). This allows the system to process neural signals as semantic representations.
- Unified Prompt Construction: The original text query is combined with the decoded brain embeddings into a single prompt. Special tokens (
<b>and</b>) are used to encapsulate the brain signals, enabling the model to distinguish between textual and neural inputs. - LLM Generation: The system utilizes LLaMA-7B as its backbone. A "prompt tuning" strategy is adopted where the LLM remains frozen, and only the brain decoder is trained. This allows the model to generate a "continuation" of the query that reflects the user's underlying thoughts.
- Ranking-oriented Inference: To optimize the generated text for retrieval, the authors incorporate Inverse Document Frequency (IDF) into the generation process. This encourages the LLM to produce specific, discriminative terms that are more effective for distinguishing target documents within a corpus.
Datasets and Experimental Setup
The framework was evaluated using three publicly available fMRI datasets involving linguistic stimuli:
- Pereira’s Dataset: Wikipedia-style sentence stimuli.
- Huth’s Dataset: Auditory stories from "The Moth Radio Hour."
- Narratives Dataset: A large-scale auditory story dataset.
To simulate a search environment, the researchers used an Inverse Cloze Test (ICT) setup. In this configuration, a specific text snippet is treated as the "query," while the surrounding document is considered the "relevant result."
Results and Discussion
Performance Gains
Across all tested datasets, Brain-Aug demonstrated significant improvements in ranking quality compared to original queries and traditional unsupervised augmentation methods. For instance, on the Pereira dataset using the BM25 model, Brain-Aug increased the NDCG@20 from 0.664 to 0.691.
The Brain as a Disambiguator
A key insight from the study is that brain signals are most effective when the initial query is highly ambiguous. For example, a query like "shaking" can be clarified by neural signals to refer specifically to an "earthquake" rather than a "cold breeze." This highlights the unique value of neural data in capturing nuances that are often lost in brief text queries.
Neural-to-Latent Mapping
The research confirms that brain signals can be mapped directly into the mathematical "thought space" (latent space) of an LLM without needing an intermediate translation into natural language. This direct mapping is more efficient and preserves semantic richness. Furthermore, the inclusion of IDF weights during inference successfully repurposed a general-purpose LLM for the specialized task of discriminative retrieval.
Conclusion and Future Work
While Brain-Aug represents a significant step toward "mind-reading" search engines, several challenges remain. The current reliance on fMRI—which is bulky and expensive—limits practical application. Preliminary tests with portable EEG showed lower signal-to-noise ratios, indicating a need for better decoding techniques for mobile neuroimaging.
Future work will focus on cross-subject generalization to eliminate the need for individual calibration and exploring real-world search scenarios beyond simulated ICT setups. Additionally, the authors emphasize the critical importance of privacy and ethical safeguards when handling sensitive neural data.
Summary updated