Neural Activation Semantic Models: Computational lexical semantic models of localized neural activations

Nikos Athanasiou, Elias Iosif, Alexandros PotamianosCOLING 2018 · OpenReview

Introduction

Traditional computational linguistics relies heavily on the distributional hypothesis: the idea that words appearing in similar textual contexts share similar meanings. While models like word2vec and GloVe have proven powerful, they lack the sensory and physical grounding inherent to human cognition. When humans process language, meaning is not just a statistical co-occurrence; it is tied to specific neural activations in sensory and motor regions of the brain.

This research explores whether localized neural activation patterns—captured via functional Magnetic Resonance Imaging (fMRI)—can be used to build superior computational semantic models. By predicting how a word "lights up" the brain, the authors aim to create "neural embeddings" that capture nuances of meaning often missed by text-only models.

Methodology

The proposed framework builds upon the foundational work of Mitchell et al. (2008) to map text-based features to brain activity. The process involves four primary stages:

  1. Voxel Selection: The model identifies "stable" voxels (3D units of brain activity) from fMRI scans of participants reading specific words.
  2. Neural Activation Prediction: Using regression, the model maps text-based semantic features (co-occurrence with 25 sensory-motor verbs) to these stable voxels. This allows the system to generate a simulated fMRI activation pattern for any concrete noun.
  3. Neural Similarity Metric: A novel weighted squared distance metric is introduced to calculate semantic similarity based on the predicted neural activation space.
  4. Multimodal Fusion: To leverage both textual and cognitive data, the authors combine neural vectors with standard embeddings (word2vec/GloVe) through early fusion (concatenation) or late fusion (similarity matrix averaging).
Figure 1: Predicted fMRI activation patterns for the concrete nouns "truck" and "car," illustrating the localized neural responses used to build the semantic model.

Experimental Setup

The models were evaluated across several diverse datasets:

  • Mitchell et al. (2008) fMRI Dataset: Brain scans of 9 participants reading 60 concrete nouns, used for training the predictor.
  • MEN: A semantic similarity dataset filtered for concrete nouns (1,114 pairs).
  • ESSLLI: Used for taxonomy creation and clustering tasks (e.g., grouping "tools" vs. "vehicles").
  • Sensicon: A dictionary mapping words to five sensory modalities (vision, audition, taste, smell, touch).
  • SNLI: The Stanford Natural Language Inference dataset, used to test performance on textual entailment.

Results and Discussion

Semantic Similarity and the "Extreme" Advantage

A key finding of this study is that neural models excel at the extremes of similarity. While word2vec performs better on general semantic similarity across the entire MEN dataset (0.76 vs. 0.48), the neural model is significantly more effective at differentiating between highly similar or highly dissimilar words.

Figure 4: Performance comparison on the MEN dataset. The neural model significantly outperforms word2vec on the "Most similar" subset, achieving a correlation of 0.83.

Taxonomy and Sense Classification

In tasks requiring categorization based on physical properties, the fused models (Neural + Text) consistently outperformed standalone text embeddings. On the ESSLLI taxonomy task, the fused model achieved a cluster purity of 0.71 for 6-class categorization. Similarly, in the Sensicon 5-way sensory classification task, the fused model reached a purity of 0.39, compared to 0.33 for word2vec. This suggests that neural embeddings capture physical and sensory nuances that are often latent or absent in text-based distributions.

Textual Entailment

The utility of neural embeddings extends to complex downstream tasks. By feeding concatenated GloVe and neural vectors into a Bi-LSTM network, the authors improved accuracy on the SNLI textual entailment task from 76.6% (GloVe only) to 77.7% on a specific subset of concrete noun pairs.

Conclusion and Future Work

This research demonstrates that cognitively grounded neural embeddings provide a unique semantic signal that complements traditional text-based models. While currently limited by the high cost and noise of fMRI data—and a focus on concrete nouns—the results indicate that "brain-inspired" representations can improve performance in sense classification and taxonomy creation. Future work will focus on lexical expansion to include abstract concepts and exploring more sophisticated fusion algorithms to better integrate these disparate semantic spaces.

Summary updated

BibTeX
@inproceedings{athanasiou2018neural,
  author = {Nikos Athanasiou and Elias Iosif and Alexandros Potamianos},
  title = {{Neural Activation Semantic Models: Computational lexical semantic models of localized neural activations}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=0dAG9Fy5j-},
}
Figure