Representation and Pre-Activation of Lexical-Semantic Knowledge in Neural Language Models

Steven Derby, Paul Miller, Barry DevereuxNAACL 2021

TL;DR

This paper investigates how well neural language models (LMs), specifically LSTMs and Transformers, capture human-like semantic knowledge. By analyzing model activations before and after a target word is processed, they find Transformers are better at capturing lexical-semantic knowledge, especially for attributive properties, and that this knowledge is retained throughout the network layers.

What Problem Does This Paper Address?

Neural language models (LMs) like LSTMs and Transformers have revolutionized Natural Language Processing (NLP) by learning rich contextual representations of words. While we know these models capture linguistic regularities, it's still a challenge to determine if and how they activate lexical-semantic knowledge in a way that mirrors human comprehension. Specifically, researchers want to know if LMs store and process semantic information similarly to how humans do, especially considering theories of human language comprehension that emphasize prediction. This paper aims to bridge this gap by systematically probing intermediate layers of LMs to see what semantic knowledge they hold, both in anticipation of a word and after it has been processed.

Methods

The authors employ a layer-by-layer analysis of LMs, focusing on two temporal modalities:

  • Before: Examining model activations just before a target word is presented. This simulates the "forward inferencing" or predictive aspect of human comprehension.
  • After: Examining model activations immediately after the target word has been input. This reflects the model's understanding once the word is available.

To achieve this, they use a "feature pooling" method. For a given target word, they extract sentences where it appears and then aggregate the hidden states from the model's intermediate layers at the timepoint before the word and at the word's position. This process aims to decontextualize the representations, focusing on the knowledge associated with the target word itself.

They then evaluate these extracted representations using three main approaches:

  1. Semantic Similarity Benchmarks: Comparing cosine similarities between word embeddings from the LMs with human similarity judgments on word pairs.
  2. BrainBench (fMRI/MEG): Evaluating how well LM representations predict human brain activity patterns associated with concept words.
  3. Human Property Knowledge (CSLB norms): Using probing techniques (logistic regression) to see if specific semantic properties (e.g., "is green," "is a fruit") can be decoded from LM embeddings.

Datasets

  • Training Data: The paper mentions using the "One Billion Word corpus" for training the JLM model. The GPT-2 model is pre-trained.
  • Evaluation Datasets:
    • Semantic Similarity: WordSim353-rel, WordSim353-sim, WordSim353, SimLex999, MEN.
  • Brain Data: BrainBench (fMRI and MEG data for 60 concept words).
  • Property Norms: CSLB norms (human-elicited semantic properties for 638 concept words).

Key Insights and Ideas

  • Transformer vs. LSTM: Transformers (specifically GPT-2) generally outperform LSTMs (JLM) in capturing semantic knowledge, particularly for attributive properties (e.g., "is round," "is green") which are often harder to model than taxonomic ones (e.g., "is a fruit").
  • "Before" vs. "After" Matters: The "after" representations (when the target word is available) consistently show better performance across all benchmarks, as expected. However, the "before" representations still capture significant semantic information, highlighting the predictive capabilities of LMs.
  • Knowledge Retention: A crucial finding is that semantic knowledge is not forgotten in later layers after the target word is processed. Instead, the models seem to retain and even improve their representation of this knowledge as information propagates through the network. This challenges the idea that only earlier layers hold explicit information about the current word.
  • Layer-Specific Roles: While the "after" representations show knowledge retention, there's a subtle distinction: earlier layers in the "after" state might reflect the just-presented word, while later layers better reflect the upcoming word's semantic content. Conversely, "before" representations show a trend where later layers are better at capturing semantic information about the to-be-predicted word.
  • Cognitive Plausibility: The JLM model (LSTM) sometimes shows better performance on brain data, which the authors suggest might be due to its more "cognitively plausible" incremental architecture.

Key Results

  • Semantic Similarity: Both models perform better with "after" representations. GPT-2 achieves stronger correlations with human similarity judgments than JLM. For JLM, the second layer ("after") performs best, while for GPT-2, the best performance varies by benchmark but is generally in later layers.
  • Brain Data (BrainBench): JLM outperforms GPT-2 in predicting brain data, especially in the "before" state. This is attributed to JLM's incremental nature.
  • Property Decoding (CSLB): GPT-2 significantly outperforms JLM in decoding semantic properties, especially attributive ones. Both models show better performance in later layers for "before" representations and upper layers for "after" representations. GPT-2's performance narrows the gap between taxonomic and attributive properties.
  • Final Layer Performance: Interestingly, the final layer of GPT-2 ("before" state) shows a consistent loss in performance across evaluation tasks, which is notable given its common use in transfer learning.

Limitations and Future Work

  • Feature Pooling Limitation: The authors acknowledge that their feature pooling method, while generally effective for static embeddings, might not fully capture the dynamic nature of word representations within a sentence, especially for transformers where representations are highly contextualized and can become dissimilar across time steps.
  • Final Layer Performance: The drop in performance for the final "before" layer of GPT-2 is a point of interest for future work, especially concerning its implications for transfer learning.
  • Further Probing: The paper suggests that future work could explore more nuanced probing methodologies and a wider range of semantic phenomena to gain a deeper understanding of LM representations.
  • Model Architectures: While comparing LSTM and Transformer, exploring other architectures or variations could yield further insights.

Summary updated

BibTeX
@inproceedings{derby2021representation,
  author = {Steven Derby and Paul Miller and Barry Devereux},
  title = {{Representation and Pre-Activation of Lexical-Semantic Knowledge in Neural Language Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}

Loading related content...

Figure