MOSAICo: a Multilingual Open-text Semantically Annotated Interlinked Corpus

Simone Conia, Edoardo Barba, Abelardo Carlos Martinez Lorenzo, Pere-Lluís Huguet Cabot, Riccardo Orlando, Luigi Procopio, Roberto NavigliNAACL 2024 · DOI

TL;DR

This paper introduces MOSAICO, a massive, multilingual corpus annotated for four key Natural Language Understanding (NLU) tasks: Word Sense Disambiguation (WSD), Semantic Role Labeling (SRL), Semantic Parsing (SP), and Relation Extraction (RE). MOSAICO aims to bridge the gap in high-quality, open-access semantic datasets, enabling researchers to develop more robust and generalizable NLU systems across languages.

What Problem Does This Paper Address?

Modern Natural Language Understanding (NLU) systems often rely on either implicitly learning semantics from vast amounts of text (like in large language models) or explicitly representing meaning using symbolic structures. While implicit methods have shown impressive results, they raise concerns about computational cost, environmental impact, and interpretability. Explicit, or "neuro-symbolic," approaches promise better interpretability and efficiency but have been severely hampered by the lack of large-scale, high-quality, multilingual datasets for tasks like WSD, SRL, SP, and RE. Existing datasets are often limited in scope, language coverage, or are proprietary, hindering progress in developing truly universal NLU capabilities.

Methods

The core contribution is the creation of the MOSAICO corpus. The authors collected data from Wikipedia across five languages (English, French, German, Italian, and Spanish), focusing on articles present in all languages to ensure comparability. They then employed state-of-the-art models for each of the four target NLU tasks to automatically generate "silver-quality" annotations.

  • Word Sense Disambiguation (WSD): The ESCHER model was used to disambiguate word senses, leveraging BabelNet as a unified sense inventory.
  • Semantic Role Labeling (SRL): The Multi-SRL system was employed to identify predicates and their arguments, using PropBank and VerbAtlas as inventories.
  • Semantic Parsing (SP): For English, the LeakDistill parser was used for Abstract Meaning Representation (AMR) graphs. For other languages, a multilingual extension of SPRING (CLAP) was utilized, trained on automatically translated English AMR graphs.
  • Relation Extraction (RE): The mREBEL system was used to extract relational triplets, with additional enrichment from Wikipedia hyperlinks and WSD annotations, leveraging Wikidata for relation types.

A subset of MOSAICO, named MOSAICO Core, was created by selecting articles deemed "good" or "featured" by the Wikipedia community, aiming for higher quality annotations.

Datasets

  • Source Data: Wikipedia (English, French, German, Italian, Spanish).
  • Introduced Benchmark: MOSAICO corpus, containing hundreds of millions of silver-quality annotations for WSD, SRL, SP, and RE across five languages. MOSAICO Core is a higher-quality subset.
  • Evaluation Benchmarks:
    • WSD: SemCor, WGC, ALL, 42D, XL-WSD.
  • SRL: OntoNotes 5.0, PB-Examples (PBE), X-SRL.
  • SP: AMR 3.0, The Little Prince (TLP), Bio AMR, AMR 2.0 – Four Translations.
  • RE: CoNLL04, NYT, ADE.

Key Insights and Ideas

  • The Power of Silver Annotations at Scale: MOSAICO demonstrates that even "silver-quality" (automatically generated) annotations, when produced at a massive scale and across multiple languages, can significantly boost the performance of state-of-the-art models, often matching or exceeding performance trained on smaller, gold-standard datasets.
  • Interconnectedness of Semantic Tasks: The paper highlights how MOSAICO's unified annotations enable the study of inter-task relationships. For instance, WSD errors can impact SRL, and the overlap between SRL and SP structures is substantial, suggesting potential for synergistic model development.
  • Bridging Inventories is Crucial: The authors found that even when systems perform well individually, direct interconnection (e.g., between WSD and SRL) reveals significant disagreement, often due to differing inventories (like BabelNet vs. VerbAtlas). This points to a future research direction in aligning these semantic formalisms.
  • Wikipedia as a Rich, Multilingual Resource: The study leverages Wikipedia's inherent multilingualism and interlinking structure to create a comprehensive semantic resource, demonstrating its potential beyond simple text retrieval.

Key Results

  • WSD: MOSAICO Core improved performance over gold-standard training data on several benchmarks, with the full MOSAICO dataset showing strong results, especially for rare senses.
  • SRL: MOSAICO Core-trained models achieved performance comparable to gold-standard datasets on OntoNotes and even surpassed them on the out-of-domain PB-Examples benchmark, particularly for challenging adjectival predicates.
  • SP: MOSAICO Core-trained models performed comparably to gold-standard English AMR parsers and showed improvements on out-of-domain and multilingual test sets.
  • RE: Models pre-trained on MOSAICO Core (which is significantly smaller than other silver corpora like REBEL) performed competitively, suggesting MOSAICO's efficiency and effectiveness for pre-training.
  • Efficiency: The authors estimate that MOSAICO can save researchers over 10,000 GPU hours.

Limitations and Future Work

  • Manual Evaluation: While proxy results confirm quality, a full manual evaluation of MOSAICO's annotations was not feasible due to its scale and multilingual nature. The authors suggest this would be a valuable future contribution.
  • Language and Task Expansion: The authors acknowledge the desire to expand MOSAICO to more languages and incorporate additional semantic tasks to further explore inter-task connections.
  • Wikipedia Biases: The use of Wikipedia introduces potential biases related to topic coverage and writing style, which might affect generalizability to other domains. However, they note that their evaluation on out-of-domain sets shows MOSAICO can still improve generalization.
  • Low-Resource Languages: Wikipedia's coverage is biased towards high-resource languages, limiting MOSAICO's depth for very low-resource languages.
  • Future Work: The paper suggests future work on bridging semantic inventories (WSD, SRL, SP), leveraging MOSAICO for cross-lingual transfer learning, and using the corpus to create new Wikidata annotations. The authors also plan to continue updating MOSAICO with new tasks and languages.

Summary updated

BibTeX
@inproceedings{conia2024mosaico,
  author = {Simone Conia and Edoardo Barba and Abelardo Carlos Martinez Lorenzo and Pere-Lluís Huguet Cabot and Riccardo Orlando and Luigi Procopio and Roberto Navigli},
  title = {{MOSAICo: a Multilingual Open-text Semantically Annotated Interlinked Corpus}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.442},
}
Figure