Multilingual Event Linking to Wikidata

Adithya Pratapa, Rishubh Gupta, Teruko Mitamura

TL;DR

This paper introduces a novel task and a large-scale dataset for multilingual event linking, connecting textual mentions of events to their corresponding entries in Wikidata across 44 languages. They demonstrate that modern neural network models significantly outperform traditional methods, though crosslingual linking remains more challenging than its monolingual counterpart.

What Problem Does This Paper Address?

Information extraction and understanding often rely on grounding textual mentions to structured knowledge bases. While entity linking (connecting mentions of people, places, organizations to knowledge bases) is well-studied, event linking (connecting mentions of actions or occurrences) is less explored, especially in a multilingual context. Existing event linking work often focuses on a single language or a limited set of languages. This paper addresses the gap by creating a comprehensive multilingual dataset and evaluating models for linking event mentions from various languages to Wikidata, a rich, multilingual knowledge base. This is crucial for building systems that can understand and reason about events across diverse linguistic sources.

Methods

The authors adapt a retrieve-and-rank approach, inspired by successful entity linking systems like BLINK. This involves two main stages:

  1. Bi-encoder (Retriever): A BERT-based model independently encodes the mention's context and candidate events. The top-k most similar events are retrieved using a similarity score (dot product of embeddings).
  2. Cross-encoder (Ranker): A more powerful model takes the concatenated context and candidate event as input to re-rank the top-k retrieved events and select the most likely match.

They experiment with multilingual versions of BERT (mBERT) and XLM-RoBERTa as the underlying encoders. For retrieval, they also include BM25 as a baseline.

Datasets

The paper introduces a new, large-scale dataset for multilingual event linking:

  • Multilingual Event Linking Dataset: Comprises 1.8 million mentions across 44 languages, linked to over 10,900 events from Wikidata. This dataset was constructed by leveraging Wikipedia hyperlinks.
  • Wikinews-based Evaluation Set: A smaller, separate dataset created from Wikinews articles to evaluate the models' out-of-domain generalization capabilities.

Key Insights and Ideas

  • Wikidata as a Multilingual Anchor: Wikidata's multilingual nature and structured properties make it an ideal knowledge base for grounding events across languages.
  • Hyperlinks as a Powerful Signal: The paper effectively uses Wikipedia's internal hyperlinks to automatically identify and link event mentions to corresponding Wikipedia pages, which are then connected to Wikidata.
  • Crosslingual Linking is Harder: The crosslingual task (linking mentions to English event descriptions) is consistently more challenging than the multilingual task (linking to descriptions in the mention's language). This suggests that language-specific nuances in event descriptions can be highly informative.
  • Temporal Reasoning is Crucial: The dataset's inclusion of event sequences highlights the importance of temporal reasoning for accurate event linking, an area where current models sometimes struggle.

Key Results

  • Neural Models Dominate: Bi-encoder and cross-encoder models based on mBERT and XLM-RoBERTa significantly outperform the BM25 baseline, achieving accuracies in the high 80s and 90s for the multilingual task.
  • Crosslingual Performance: While still strong, crosslingual performance is generally lower than multilingual performance, with accuracies in the 70s and 80s.
  • Wikinews Generalization: Models show good cross-domain generalization to Wikinews but moderate zero-shot transfer, indicating that unseen domains and events remain a challenge. Including article titles and dates in the context further improves performance on Wikinews.

Limitations and Future Work

  • Event Type Focus: The current dataset primarily focuses on "eventive nouns" due to the way events are identified in Wikidata. Expanding to include more verbal events is a future direction.
  • Temporal Reasoning Gaps: The paper acknowledges that temporal reasoning remains a challenge, especially for event sequences.
  • Hierarchical Event Structures: The authors suggest expanding the event dictionary to include hierarchical event structures in Wikidata for more complex linking tasks.
  • Language-Specific Descriptions: The benefit of language-specific event descriptions in the multilingual task suggests exploring how to better leverage these for crosslingual linking.
  • Dataset Errors: While efforts were made to clean the data, some instances of incorrect hyperlinks or insufficient context were found, indicating potential for further dataset refinement.

Summary updated

Abstract

We present a task of multilingual linking of events to a knowledge base. We automatically compile a large-scale dataset for this task, comprising of 1.8M mentions across 44 languages referring to over 10.9K events from Wikidata. We propose two variants of the event linking task: 1) multilingual, where event descriptions are from the same language as the mention, and 2) crosslingual, where all event descriptions are in English. On the two proposed tasks, we compare multiple event linking systems including BM25+ (Lv and Zhai, 2011) and multilingual adaptations of the biencoder and crossencoder architectures from BLINK (Wu et al., 2020). In our experiments on the two task variants, we find both biencoder and crossencoder models significantly outperform the BM25+ baseline. Our results also indicate that the crosslingual task is in general more challenging than the multilingual task. To test the out-of-domain generalization of the proposed linking systems, we additionally create a Wikinews-based evaluation set. We present qualitative analysis highlighting various aspects captured by the proposed dataset, including the need for temporal reasoning over context and tackling diverse event descriptions across languages.
BibTeX
@inproceedings{pratapa2022multilingual,
  author = {Adithya Pratapa and Rishubh Gupta and Teruko Mitamura},
  title = {{Multilingual Event Linking to Wikidata}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure