Towards brain-activity-controlled information retrieval: Decoding image relevance from MEG signals

Jukka-Pekka Kauppi, Melih Kandemir, Veli-Matti Saarinen, Lotta Hirvenkari, Lauri Parkkonen, Arto Klami, Riitta Hari, Samuel Kaski

TL;DR

This paper explores the feasibility of using brain activity (MEG signals) to infer image relevance for information retrieval systems, moving beyond traditional explicit user feedback. They demonstrate that MEG signals alone can predict image relevance better than chance, and that combining MEG with gaze data further improves prediction accuracy, paving the way for more intuitive brain-computer interfaces for search.

What Problem Does This Paper Address?

Interactive information retrieval (IIR) systems typically rely on users explicitly providing feedback (e.g., rating search results) to refine their queries. This can be tedious and interrupt the user's workflow. While implicit feedback methods using behavioral signals like mouse movements and gaze tracking have shown promise, they may not capture the full picture of user intent. This paper addresses the gap of using brain activity, specifically magnetoencephalography (MEG) signals, as a richer source of implicit relevance feedback for IIR systems. The core challenge is to decode subtle brain signals related to visual attention and relevance judgments from noisy, real-time data during naturalistic visual exploration.

Methods

The researchers employed a multi-modal approach, combining MEG and eye-tracking data.

  • MEG Signal Processing: They focused on short epochs of MEG data time-locked to when a subject fixated on an image. To extract relevant features, they applied band-pass filters to isolate different frequency bands (delta, theta, alpha, beta, gamma) within these epochs. A novel "Spectral PCA" approach was used to reduce the dimensionality of these spectral features by applying Principal Component Analysis (PCA) channel-wise. This method aimed to capture spectral information specific to each brain region while preventing overfitting.
  • Gaze Signal Processing: Standard eye-tracking features, including those related to fixation duration, saccade patterns, and pupil diameter, were extracted.
  • Classification: A Gaussian Process (GP) classifier with a Radial Basis Function (RBF) kernel was used for decoding. GPs are non-parametric Bayesian models that provide probabilistic outputs and uncertainty estimates, making them suitable for noisy neuroimaging data and enabling "late fusion" of different modalities. They compared this non-linear classifier to a Bayesian linear logistic regression (LR) model.
  • Fusion: A "late fusion" strategy was employed, where the predictions from separate MEG and gaze classifiers were combined (averaged) to produce a final relevance prediction.

Datasets

The study used data collected from nine human participants. The participants viewed image collages, where a subset of images was relevant to a specific task (e.g., "find a cheetah"). The ground truth relevance of each image was known based on the task definition. No new benchmark datasets were introduced; instead, they utilized a free-viewing setup that more closely mimics real-world usage compared to highly controlled experimental designs.

Key Insights and Ideas

  • Brain Activity as a Relevance Signal: The most significant insight is that brain activity, as measured by MEG, contains information that can reliably predict the relevance of visual stimuli, even during naturalistic free viewing. This moves beyond solely relying on overt behaviors.
  • Synergy of MEG and Gaze: Combining MEG and gaze signals significantly improves relevance prediction compared to using either modality alone. This suggests that these signals capture complementary information about user attention and intent.
  • Non-linear Models are Crucial: The use of a non-linear Gaussian Process classifier outperformed linear models, highlighting that the brain's response to relevance is not always linearly separable in the measured signals.
  • Spectral PCA for Feature Extraction: The proposed Spectral PCA method for MEG feature extraction proved effective, outperforming simpler feature sets like raw band power or concatenated band powers. This suggests a sophisticated way to handle the high dimensionality and spectral richness of MEG data.

Key Results

  • MEG-only Performance: The GP classifier using MEG signals alone achieved significantly better-than-chance performance for most subjects, with a mean AUC-ROC of 0.654 and AUC-PR of 0.348.
  • Gaze-only Performance: The gaze-based classifier performed comparably, with a mean AUC-ROC of 0.655 and AUC-PR of 0.363.
  • Combined Performance: Late fusion of MEG and gaze signals yielded the best results, with a mean AUC-ROC of 0.679 and AUC-PR of 0.386. This combined performance was statistically significantly better than the gaze-only classifier for both metrics.
  • GP vs. Linear: The GP classifier significantly outperformed the Bayesian linear logistic regression model for both AUC-ROC (0.654 vs. 0.609) and AUC-PR (0.348 vs. 0.270).
  • Feature Importance: The study demonstrated the ability to visualize the contribution of different MEG channels to prediction, revealing spatial patterns of brain activity associated with relevance judgments.

Limitations and Future Work

  • MEG Signal Noise and Artifacts: While efforts were made to mitigate eye artifacts by focusing on posterior MEG channels, the inherent noisiness of MEG signals remains a challenge.
  • Limited Number of Subjects: The study was conducted with nine subjects, and while results were statistically significant, a larger sample size would increase generalizability.
  • "Objective" vs. "Subjective" Relevance: The study used "objective" relevance labels (ground truth based on task content) rather than "subjective" user-defined relevance. While this simplifies labeling, future work could explore how subjective relevance can be inferred.
  • Temporal Features: The current study focused on spectral features. The authors suggest that exploring temporal features of MEG signals could potentially offer additional discriminative information, though this would require a different experimental design due to the short epoch lengths.
  • Real-time Application: While the methods show promise, the computational demands of GP classification and feature extraction might need optimization for true real-time IIR applications.

The authors envision this work as a foundational step towards brain-activity-based interactive information retrieval systems, suggesting that future systems could integrate both gaze and brain activity for more intuitive and effective search experiences.

Summary updated

Abstract

We hypothesize that brain activity can be used to control future information retrieval systems. To this end, we conducted a feasibility study on predicting the relevance of visual objects from brain activity. We analyze both magnetoencephalographic (MEG) and gaze signals from nine subjects who were viewing image collages, a subset of which was relevant to a predetermined task. We report three findings: i) the relevance of an image a subject looks at can be decoded from MEG signals with performance significantly better than chance, ii) fusion of gaze-based and MEG-based classifiers significantly improves the prediction performance compared to using either signal alone, and iii) non-linear classification of the MEG signals using Gaussian process classifiers outperforms linear classification. These findings break new ground for building brain-activity-based interactive image retrieval systems, as well as for systems utilizing feedback both from brain activity and eye movements.
BibTeX
@article{kauppi2015towards,
  author = {Jukka-Pekka Kauppi and Melih Kandemir and Veli-Matti Saarinen and Lotta Hirvenkari and Lauri Parkkonen and Arto Klami and Riitta Hari and Samuel Kaski},
  title = {{Towards brain-activity-controlled information retrieval: Decoding image relevance from MEG signals}},
  journal = {NeuroImage},
  year = {2015},
  doi = {10.1016/j.neuroimage.2014.12.079},
}
Figure