Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model

Wenhan Xiong, Jingfei Du, William Yang Wang, Veselin Stoyanov

Abstract: Recent breakthroughs of pretrained language models have shown the effectiveness of self-supervised learning for a wide range of natural language processing (NLP) tasks. In addition to standard syntactic and semantic NLP tasks, pretrained models achieve strong improvements on tasks that involve real-world knowledge, suggesting that large-scale language modeling could be an implicit method to capture knowledge. In this work, we further investigate the extent to which pretrained models such as BERT capture knowledge using a zero-shot fact completion task. Moreover, we propose a simple yet effective weakly supervised pretraining objective, which explicitly forces the model to incorporate knowledge about real-world entities. Models trained with our new objective yield significant improvements on the fact completion task. When applied to downstream tasks, our model consistently outperforms BERT on four entity-related question answering datasets (i.e., WebQuestions, TriviaQA, SearchQA and Quasar-T) with an average 2.7 F1 improvements and a standard fine-grained entity typing dataset (i.e., FIGER) with 5.7 accuracy gains.

Introduction

While large-scale language models like BERT and GPT-2 are masters of syntax and semantics, their knowledge of the real world is largely implicit. They learn that "Paris" often appears near "France," but they do not explicitly model entities as distinct, interconnected concepts. Standard training objectives like Masked Language Modeling (MLM) operate at the token level, which often fails to capture the complex relations inherent in multi-token entities. Furthermore, previous attempts to inject knowledge into models often required complex architectural changes or massive external knowledge base lookups that made the models heavy and difficult to use.

To address this, researchers developed WKLM (Weakly Supervised Knowledge-Pretrained Language Model), which explicitly learns real-world facts by playing a "spot the imposter" game with entities during pretraining.

The WKLM Approach

The core innovation of WKLM is a pretraining objective called Entity Replacement Training (ERT). Instead of just predicting a masked word, the model learns by distinguishing truth from plausible-sounding falsehoods.

Overview of Entity Replacement Training (ERT) comparing original and replaced articles.

In this framework, the model is presented with a text where some entity mentions have been replaced by "imposters"—other entities of the same type. The model's task is a binary classification: determining if an entity mention in a given context is the original or a replacement.

To ensure the task requires factual knowledge rather than just detecting syntactic anomalies, the replacements are type-constrained. For instance, replacing "Marvel Comics" with "DC Comics" (both publishing companies) keeps the sentence "Spider-Man is published by..." grammatically correct.

The imposter strategy: using Wikidata to find type-consistent entity replacements.

As shown above, the process utilizes Wikipedia anchor links and Wikidata's knowledge graph to identify entities and their types. By choosing a replacement of the same type, the sentence remains linguistically plausible. To tell the difference between "Spider-Man was created by Stan Lee" and "Spider-Man was created by Bryan Johnson," the model must move beyond simple grammar and actually learn encyclopedic knowledge.

Experimental Evaluation

WKLM was pretrained on the entire English Wikipedia dump, cross-referenced with Wikidata for entity types and aliases. The model was evaluated across several knowledge-intensive benchmarks:

  • Fact Completion: A custom set of 1,000 cloze examples for 10 common Wikidata relations (e.g., CapitalOf, FoundedBy).
  • Entity Typing: The FIGER dataset, which requires labeling entities with fine-grained types (e.g., /person/author).
  • Open-Domain QA: Four major datasets: WebQuestions, TriviaQA, SearchQA, and Quasar-T.
  • Reading Comprehension: SQuAD, used to validate that the model still performs well on standard NLP tasks.

Key Results

WKLM demonstrated significant improvements across all entity-related tasks while maintaining general linguistic prowess:

  • Fact Completion: WKLM achieved a 28.9 Hits@10 score, nearly doubling the performance of BERT-large (16.1) and significantly outperforming GPT-2 (16.3).
  • Entity Typing: On the FIGER dataset, WKLM achieved a 5.7% accuracy gain over previous methods, setting a new state-of-the-art.
  • Question Answering: Across four entity-heavy QA datasets, WKLM provided an average 2.7 F1 improvement over the BERT baseline.
  • Efficiency: Unlike models that require constant lookups to an external database, WKLM stores the knowledge in its parameters, making it as fast and easy to use as a standard BERT model during inference.

Conclusion

WKLM proves that a simple, weakly supervised objective can effectively bake encyclopedic knowledge into language models. By forcing the model to distinguish between facts and plausible falsehoods, Entity Replacement Training creates a more robust representation of real-world entities. This approach allows the model to retain its general language understanding while significantly benefiting downstream knowledge-intensive tasks like question answering and entity typing.

Summary updated

Abstract

Recent breakthroughs of pretrained language models have shown the effectiveness of self-supervised learning for a wide range of natural language processing (NLP) tasks. In addition to standard syntactic and semantic NLP tasks, pretrained models achieve strong improvements on tasks that involve real-world knowledge, suggesting that large-scale language modeling could be an implicit method to capture knowledge. In this work, we further investigate the extent to which pretrained models such as BERT capture knowledge using a zero-shot fact completion task. Moreover, we propose a simple yet effective weakly supervised pretraining objective, which explicitly forces the model to incorporate knowledge about real-world entities. Models trained with our new objective yield significant improvements on the fact completion task. When applied to downstream tasks, our model consistently outperforms BERT on four entity-related question answering datasets (i.e., WebQuestions, TriviaQA, SearchQA and Quasar-T) with an average 2.7 F1 improvements and a standard fine-grained entity typing dataset (i.e., FIGER) with 5.7 accuracy gains.
BibTeX
@inproceedings{xiong2020pretrained,
  author = {Wenhan Xiong and Jingfei Du and William Yang Wang and Veselin Stoyanov},
  title = {{Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=LEMFZn7Ndv},
}
Figure