Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model

Wenhan Xiong, Jingfei Du, William Yang Wang, Veselin Stoyanov

TL;DR

Researchers have developed a new pretraining method called WKLM (Weakly Supervised Knowledge-Pretrained Language Model) that teaches AI to distinguish between real-world facts and plausible-sounding fakes. By training a model to "spot the difference" when entities are swapped with others of the same type, the authors significantly improved the model's performance on fact-heavy tasks like open-domain question answering and entity typing.

What Problem Does This Paper Address?

While large-scale language models like BERT are masters of grammar and syntax, their "knowledge" of the real world is often incidental and shallow. They learn that "Paris" often appears near "France," but they do not explicitly model entities as distinct units of knowledge.

Existing models struggle with "zero-shot fact completion"—predicting a missing entity in a sentence like "The capital of France is [MASK]"—especially when the answer involves multiple words. The challenge is that standard training (Masked Language Modeling) treats every word the same. This paper addresses the gap between linguistic fluency and encyclopedic knowledge by forcing the model to pay closer attention to real-world entities and their relationships.

Methods

The authors introduce a novel pretraining objective called Type-Constrained Entity Replacement. Here is how the framework operates:

  1. Identify Entities: They take a massive corpus (English Wikipedia) and use anchor links to identify real-world entities.
  2. The "Fake News" Generator: For a given sentence, they randomly replace an entity mention with another entity of the same type from Wikidata.
Entity replacement process using Wikidata types

As shown above, an entity like "Marvel Comics" is linked to its Wikidata type (book publishing company). The system then samples another entity from that same cluster (e.g., "DC Comics") to use as a replacement. In a sentence like "Spider-Man was created by Stan Lee," they might replace "Stan Lee" with "Bryan Johnson" (another person) or "Marvel Comics" with "DC Comics" (another publisher).

  1. The Objective: Unlike BERT, which tries to guess a hidden word, WKLM is trained to perform a binary classification: Is this entity the original, or has it been replaced?
WKLM training objective comparing original and replaced articles
  1. Multi-task Learning: They combine this entity-replacement task with a modified version of BERT’s Masked Language Modeling (MLM). Crucially, they reduce the masking ratio to 5% (down from BERT's 15%) to ensure the model has enough surrounding context to judge the factual accuracy of the entities.

By swapping entities of the same type, the sentence remains grammatically perfect but becomes factually incorrect. This forces the model to move beyond linguistic patterns and actually "learn" the facts.

Datasets

  • Pretraining: The entire English Wikipedia dump, cross-referenced with Wikidata for entity types and aliases.
  • Fact Completion: A custom set of 1,000 cloze examples for each of 10 common Wikidata relations (e.g., CapitalOf, FoundedBy).
  • Downstream QA: Four popular open-domain question-answering datasets: WebQuestions, TriviaQA, SearchQA, and Quasar-T.
  • Entity Typing: The FIGER dataset, which requires the model to assign fine-grained types (like /person/author) to entity mentions in context.

Key Insights and Ideas

  • Linguistic Correctness vs. Factual Truth: Standard language models are often "fooled" by sentences that are grammatically correct but factually wrong. By using type-constrained replacements, the authors created a training signal that specifically targets factual knowledge without providing "easy" linguistic clues (like a grammar error) that the model could use as a shortcut.
  • Boundary Representations: Instead of just looking at the tokens inside an entity, the model uses the representations of the words immediately before and after the entity. This focuses the model's attention on how the context relates to the entity.
  • The "Left-Context" Bias: The authors found that BERT often relies heavily on left-to-right patterns. WKLM's bidirectional approach and entity-focus help it perform better on relations where the answer isn't just a predictable "end of the sentence" word.

Key Results

  • Fact Completion: WKLM achieved an average Hits@10 of 28.9, nearly doubling the performance of BERT-base (11.3) and significantly outperforming GPT-2 (16.3).
  • Question Answering: On entity-related QA datasets, WKLM consistently outperformed BERT, showing an average improvement of 2.7 F1 points.
  • Entity Typing: On the FIGER dataset, WKLM set a new state-of-the-art with 60.21% accuracy, a massive 5.7 point jump over previous BERT baselines.
  • Efficiency: Unlike other "knowledge-enhanced" models that require massive external knowledge graph embeddings or complex architectural changes, WKLM uses the standard Transformer architecture, making it easy to fine-tune for any downstream task.

Limitations and Future Work

  • Reliance on Linking: The current method relies on Wikipedia anchor links and Wikidata's "instance of" relations. If an entity isn't in these databases, the model won't benefit from the specialized training.
  • Entity Coverage: The authors noted that while the method is effective, it currently only uses a subset of Wikidata relations. Future work could involve scaling this to more diverse types of knowledge.
  • Unstructured Extension: While this work used Wikipedia, the authors suggest the framework could be extended to any large corpus using off-the-shelf entity linking tools, potentially allowing the model to learn specialized knowledge from medical or legal documents.

Summary updated

Abstract

Recent breakthroughs of pretrained language models have shown the effectiveness of self-supervised learning for a wide range of natural language processing (NLP) tasks. In addition to standard syntactic and semantic NLP tasks, pretrained models achieve strong improvements on tasks that involve real-world knowledge, suggesting that large-scale language modeling could be an implicit method to capture knowledge. In this work, we further investigate the extent to which pretrained models such as BERT capture knowledge using a zero-shot fact completion task. Moreover, we propose a simple yet effective weakly supervised pretraining objective, which explicitly forces the model to incorporate knowledge about real-world entities. Models trained with our new objective yield significant improvements on the fact completion task. When applied to downstream tasks, our model consistently outperforms BERT on four entity-related question answering datasets (i.e., WebQuestions, TriviaQA, SearchQA and Quasar-T) with an average 2.7 F1 improvements and a standard fine-grained entity typing dataset (i.e., FIGER) with 5.7 accuracy gains.
BibTeX
@inproceedings{xiong2020pretrained,
  author = {Wenhan Xiong and Jingfei Du and William Yang Wang and Veselin Stoyanov},
  title = {{Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=LEMFZn7Ndv},
}

Loading related content...

Figure