Representation of Word Meaning in the Intermediate Projection Layer of a Neural Language Model

Steven Derby, Paul Miller, Brian Murphy, Barry DevereuxEMNLP 2018

TL;DR

This paper investigates how state-of-the-art neural language models, specifically LSTMs, represent word meaning. By analyzing activation patterns within the model, they found that these models encode semantic knowledge that aligns with human judgments of similarity, property knowledge, and even brain imaging data, suggesting a deeper understanding of language than previously assumed.

What Problem Does This Paper Address?

Modern neural language models, particularly those based on Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) cells, have achieved remarkable performance in tasks like predicting the next word in a sequence. However, their complex, "black box" nature makes it difficult to understand how they achieve this performance. Specifically, there's a gap in understanding how these models represent the meaning of words (lexical semantics), beyond just their syntactic roles. This lack of interpretability hinders principled development of even better models and our understanding of how machines learn language.

Methods

The researchers delve into the internal workings of an LSTM language model (lm_1b). They extract activation patterns from an intermediate projection layer for specific words. To capture word meaning independent of immediate context, they average these activation patterns across 100 sentences where the word appears. They also analyze activations from the LSTM layer before the target word is processed, hypothesizing this might capture predictive semantic information. These extracted "model vectors" are then compared against human-derived semantic knowledge.

Datasets

  • lm_1b: A large-scale LSTM language model used for extracting internal representations.
  • One Billion Word dataset: Used to construct the training data for the lm_1b model.
  • WordSim353: A dataset of word pairs with human ratings for semantic similarity and relatedness, used to evaluate how well model representations predict human judgments.
  • CSLB norms (Devereux et al., 2014): A dataset of human-elicited property knowledge for concepts (e.g., "leaf has properties is-green & grows-on-trees"), used to test if model representations can predict these properties.
  • Brain-Bench (Xu et al., 2016): A dataset containing fMRI and MEG brain imaging data for 60 concepts, used to compare model representations with neural activity.

Key Insights and Ideas

  • LSTMs Encode Semantic Knowledge: The study provides strong evidence that LSTMs don't just learn syntax; they also capture significant lexical semantic information.
  • "After" Model Better for Similarity: Representations derived from activations after a word is processed correlate better with human similarity judgments, which makes intuitive sense as the word has directly influenced the model's state.
  • "Before" Model Shows Predictive Power: Surprisingly, the "before" model (activations preceding the word) still shows a notable correlation with human similarity ratings, suggesting LSTMs can anticipate upcoming semantic content.
  • Property Prediction Success: Model representations can predict human-derived properties of concepts, indicating a structured semantic representation within the model.
  • "Before" Model Better for Properties: Interestingly, semantic features were more decodable from the "before" model activations than the "after" model when predicting properties, hinting at a predictive semantic encoding.
  • Correlation with Brain Data: Model representations show a correlation with human brain imaging data (fMRI and MEG), suggesting a convergence between how LSTMs and humans represent word meaning.

Key Results

  • Similarity Judgments: The "after" model achieved a correlation of r=0.30 with human semantic similarity ratings, outperforming the "before" model (r=0.21). Both models showed stronger correlations with semantic similarity than semantic relatedness.
  • Property Knowledge Prediction: Semantic features were more decodable from the "before" model than the "after" model, though both performed above chance.
  • Brain Imaging Data: Both "before" and "after" models performed above chance when compared to fMRI and MEG data, indicating a correlation with human brain representations. The "before" model performed slightly better on fMRI data.

Limitations and Future Work

  • Context Independence Assumption: The method of averaging activations over 100 sentences to achieve context independence is an assumption. While reasonable, it might not fully capture the nuanced contextual variations in word meaning.
  • Focus on Intermediate Layer: The study focused on a specific intermediate projection layer. Future work could explore other layers or different architectural components of LSTMs.
  • Limited Scope of Concepts: While 60 concepts were used for brain imaging comparison, a broader range of concepts could provide more robust findings.
  • Future Work: The authors suggest that the ability of language models to predict human property knowledge opens avenues for direct comparisons between model activations and human language comprehension processes. Further research could explore how these semantic representations evolve across different layers and model architectures.

Summary updated

BibTeX
@inproceedings{derby2018representation,
  author = {Steven Derby and Paul Miller and Brian Murphy and Barry Devereux},
  title = {{Representation of Word Meaning in the Intermediate Projection Layer of a Neural Language Model}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure