Inspecting the concept knowledge graph encoded by modern language models

Carlos Aspillaga, Marcelo Mendoza, Alvaro SotoACL 2021

TL;DR

This paper investigates how well modern language models (like BERT, GPT-2, T5) encode conceptual knowledge, specifically the relationships found in WordNet. Using a novel "probing" technique, they found that while these models do capture a significant amount of semantic information, they also exhibit notable inaccuracies and biases, with different model architectures learning different things.

What Problem Does This Paper Address?

The incredible success of large language models (LLMs) in various Natural Language Processing (NLP) tasks has led researchers to wonder: what exactly are these models learning? While we know they excel at syntactic structures (grammar, sentence formation), their grasp of semantic meaning – the actual concepts and their relationships – remains less clear. Previous attempts to probe this semantic understanding have yielded mixed or contradictory results. This paper aims to systematically investigate how well different LLM architectures encode the rich, hierarchical knowledge of concepts and their relationships as organized in WordNet, a human-curated lexical database. Understanding these capabilities and limitations is crucial for building more robust and truly intelligent AI systems.

Methods

The core of this research is a probing classifier. Imagine training a simple model to perform a specific task using the internal "representations" (embeddings) generated by a pre-trained LLM. If this simple probe can solve the task well, it suggests the LLM has encoded the necessary information for that task.

Here's how they applied it:

  • Concept Relatedness Probe: They focused on predicting whether two concepts from WordNet are related (e.g., "dog" and "animal") or not.
  • WordNet as Ground Truth: They used the hyponymy relations (is-a relationships) in WordNet as their target knowledge. For instance, "poodle" is a hyponym of "dog," and "dog" is a hyponym of "animal."
  • Triplet Sampling: To train the probe, they created triplets of concepts (A, B, C) where B is a hypernym (parent) of A, and C is unrelated to both. This allowed them to create both positive (related) and negative (unrelated) examples for the probe.
  • Model Embeddings: For each concept, they extracted its embedding from the LLM. For contextual models (like BERT), they fed sentences containing the concept into the model and took the embedding of the specific word.
  • Classifier Architecture: The extracted embeddings were projected into a lower-dimensional space and then fed into a Multi-Layer Perceptron (MLP) classifier. Crucially, the LLM weights were kept frozen; only the MLP was trained.
  • Graph Reconstruction: Beyond just predicting pairs, they also attempted to reconstruct parts of the WordNet graph using the probe's outputs and evaluated the quality of these reconstructed graphs.

Datasets

  • WordNet: The primary dataset used for defining concepts and their relationships. Specifically, they used the Princeton WordNet Gloss Corpus for synset glosses and hyponymy relations.
  • OpenWebText Corpus: Mentioned as the source for computing concept frequencies, though not directly used for training the probe itself.

Key Insights and Ideas

  • Models Encode Semantics, But Imperfectly: All tested models show a significant ability to encode the hierarchical relationships present in WordNet. This is a fundamental finding, confirming that LLMs are not just statistical parrots but do learn conceptual structures.
  • Architecture Matters for Bias: Different model families (e.g., contextual vs. non-contextual, Transformer-based vs. others) exhibit distinct biases in how they encode semantic knowledge. This suggests that architectural choices significantly influence what kind of semantic understanding is prioritized.
  • Concept Specificity and Frequency are Key: The performance of models degrades for more specific or less frequent concepts. This makes intuitive sense: common, high-level concepts appear in more diverse contexts, making them easier to learn.
  • "Taxonomic Groups" are Tricky: All models struggled with concepts related to "taxonomic groups" (e.g., categories of categories). The authors hypothesize that the flexibility of language allows for easy substitution of such terms, confusing the models.
  • Location of Knowledge Varies: Some models (like BERT) encode semantic knowledge primarily in their top layers, while others (like XLNet) use earlier layers. This has implications for how models are fine-tuned.

Key Results

  • Consistent Performance within Families: Models within the same family (e.g., BERT, RoBERTa, ALBERT) showed similar performance levels.
  • Contextual Models Outperform Non-Contextual: Contextualized embeddings (CE) generally outperformed non-contextualized ones (NCE) and generative language models (GLM) in capturing semantic relationships, especially when considering all layers.
  • Graph Reconstruction Quality: Graphs reconstructed using CE models were closer to the ground truth WordNet graph than those from NCE and GLM models, with BERT, ALBERT, and XLNet showing particularly strong results.
  • Specific Concept Difficulties: Models consistently showed lower F1-scores for concepts that are more specific (deeper in the WordNet graph), more frequent, or belong to categories like "taxonomic groups."
  • Layer-wise Contributions: BERT and RoBERTa leverage their top layers for semantic encoding, while XLNet and ALBERT utilize earlier layers. GPT-2 uses its layers more evenly, and T5's encoder plays a crucial role.

Limitations and Future Work

  • Imprecisions Remain: Despite encoding significant knowledge, all models exhibit inaccuracies. The paper highlights the need for models that capture concepts more precisely.
  • Probing Limitations: While the probing classifier is a powerful tool, it's still a simplified task. The authors acknowledge that success on this specific task doesn't guarantee perfect semantic understanding in all real-world scenarios.
  • Focus on Hyponymy: The study primarily focused on hyponymy relations. Future work could explore other types of semantic relationships.
  • Improving Models: The findings suggest directions for future research, such as developing architectures that better handle concept specificity and frequency, or exploring ensemble methods that combine the strengths of different model families. The observation that some models encode knowledge in earlier layers could inform more efficient fine-tuning strategies.

Summary updated

BibTeX
@inproceedings{aspillaga2021inspecting,
  author = {Carlos Aspillaga and Marcelo Mendoza and Alvaro Soto},
  title = {{Inspecting the concept knowledge graph encoded by modern language models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure