Multilingual Event Linking to Wikidata

Adithya Pratapa, Rishubh Gupta, Teruko Mitamura

TL;DR

This paper introduces a new task: multilingual event linking to Wikidata, which involves connecting mentions of events in various languages to their corresponding entries in a knowledge base. They've created a large dataset for this task and show that advanced neural models significantly outperform traditional methods, though crosslingual linking remains a significant challenge.

What Problem Does This Paper Address?

Information extraction systems often struggle to understand and connect events described across different languages. While entity linking (connecting names to knowledge base entries) is well-studied, event linking – which requires considering participants, temporal, and spatial attributes – is less explored, especially in a multilingual context. Existing methods often focus on English, leaving a gap in understanding events globally. This paper aims to bridge this gap by creating a multilingual event linking task and dataset, enabling models to ground event mentions from diverse languages to a unified knowledge base (Wikidata).

Methods

The authors adapt the "retrieve+rank" pipeline, popularized by the BLINK system for entity linking, for event linking. This involves two stages:

  1. Retriever (Bi-encoder): A bi-encoder model (using mBERT or XLM-RoBERTa) encodes both the mention's context and candidate event descriptions. It then scores potential event matches using a dot product of their embeddings, retrieving the top-k candidates.
  2. Ranker (Cross-encoder): A cross-encoder model takes the mention context and a candidate event description as input and outputs a final score to re-rank the top-k candidates from the retriever, identifying the most likely event.

They also use BM25, a traditional TF-IDF based retrieval method, as a baseline.

Datasets

  • Newly Introduced Dataset: The core contribution is a large-scale, multilingual dataset for event linking. It comprises 1.8 million mentions across 44 languages that refer to over 10,900 events from Wikidata. This dataset was compiled by leveraging hyperlinks within Wikipedia articles and connecting them to Wikidata.
  • Wikinews Evaluation Set: To test out-of-domain generalization, a smaller evaluation set was created using Wikinews articles, providing a more realistic scenario for real-world applications.

Key Insights and Ideas

  • Wikidata as an Interlingua: Using Wikidata as the knowledge base is crucial because it acts as a common ground (interlingua) that connects mentions from numerous languages.
  • Crosslingual is Harder: The crosslingual task (linking mentions to English event descriptions) is consistently more challenging than the multilingual task (linking to descriptions in the same language as the mention). This suggests that language-specific nuances in event descriptions are highly informative.
  • Neural Models Outperform Baselines: Advanced bi-encoder and cross-encoder models significantly outperform the traditional BM25 baseline, demonstrating the power of deep learning for this task.
  • Temporal Reasoning is Key: The dataset's structure, with many event sequences, highlights the importance of temporal reasoning for accurate event linking, an area where current systems sometimes struggle.
  • Diverse Mention Types: Event mentions can be more varied than entity mentions, including temporal or spatial expressions, adding complexity to the linking task.

Key Results

  • Accuracy Gains: Bi-encoder and cross-encoder models achieved significantly higher accuracy than the BM25 baseline on both multilingual and crosslingual tasks. For instance, on the multilingual task, mBERT-cross achieved ~90% accuracy, while BM25+ was around 50%.
  • Crosslingual Performance: While still outperforming BM25, crosslingual performance was generally lower than multilingual performance across all models. For example, mBERT-cross achieved ~76% accuracy on the crosslingual task compared to ~90% on the multilingual task.
  • Wikinews Generalization: The models showed good cross-domain performance on the Wikinews set but moderate zero-shot transfer, indicating that unseen domains and events remain a challenge. Including article dates and titles in the context improved performance by 4-12%.

Limitations and Future Work

  • Dataset Scope: The current dataset primarily focuses on "eventive nouns" and might not fully cover verbal events.
  • Out-of-Domain Challenges: The Wikinews evaluation highlighted that generalizing to new domains and unseen events is difficult.
  • Temporal/Spatial Reasoning: Systems still struggle with temporal and spatial expressions, and distinguishing between events in sequences.
  • Future Directions:
    • Expanding the event dictionary to include hierarchical event structures.
  • Increasing mention coverage to include more verbal events.
  • Improving temporal reasoning and handling of temporal/spatial expressions.
  • Incorporating events that lack English descriptions.
  • Investigating performance differences between planned (e.g., sports) and unplanned (e.g., wars) events.

Summary updated

Abstract

We present a task of multilingual linking of events to a knowledge base. We automatically compile a large-scale dataset for this task, comprising of 1.8M mentions across 44 languages referring to over 10.9K events from Wikidata. We propose two variants of the event linking task: 1) multilingual, where event descriptions are from the same language as the mention, and 2) crosslingual, where all event descriptions are in English. On the two proposed tasks, we compare multiple event linking systems including BM25+ (Lv and Zhai, 2011) and multilingual adaptations of the biencoder and crossencoder architectures from BLINK (Wu et al., 2020). In our experiments on the two task variants, we find both biencoder and crossencoder models significantly outperform the BM25+ baseline. Our results also indicate that the crosslingual task is in general more challenging than the multilingual task. To test the out-of-domain generalization of the proposed linking systems, we additionally create a Wikinews-based evaluation set. We present qualitative analysis highlighting various aspects captured by the proposed dataset, including the need for temporal reasoning over context and tackling diverse event descriptions across languages.
BibTeX
@inproceedings{pratapa2022multilingual,
  author = {Adithya Pratapa and Rishubh Gupta and Teruko Mitamura},
  title = {{Multilingual Event Linking to Wikidata}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure