TL;DR
The authors introduce MegaWika 2, a significantly expanded multilingual dataset of Wikipedia articles, their citations, and scraped web sources. This dataset is crucial for advancing research in areas like report generation, fact-checking, and multilingual NLP, offering a richer, more structured representation of knowledge than its predecessor.
What Problem Does This Paper Address?
The paper addresses the need for comprehensive, multilingual datasets to support advanced AI tasks, particularly in report generation and fact-checking. While Wikipedia is a rich source of information, existing datasets often lack detailed structural information, fine-grained links between claims and sources, and multilingual coverage. This makes it difficult to train and evaluate models that can understand, verify, and generate reports based on complex, multilingual information. The authors aim to bridge this gap by providing MegaWika 2, a dataset that captures Wikipedia's structure, links claims to their sources with precise offsets, and supports a wide range of languages.
Methods
MegaWika 2 is built through a four-pipeline data processing system:
- Extraction: Downloads Wikipedia dumps, parses Wikitext to identify articles, citations, and structure (headings, paragraphs, tables, etc.), and scrapes web sources for URL-based citations.
- Translation: Translates article headings and sentences into English using NLLB-200, improving upon MegaWika 1's translation methods.
- Source Quality: Extracts source texts and estimates their quality using a predicted ordinal score, helping users filter out low-quality or irrelevant sources.
- Enrichment: Fetches cross-lingual article links and article creation dates, and extracts editor-provided quotes from citations.
The dataset is structured into JSON-lines files, organized by language, with each article represented as a JSON object containing its title, Wikitext, aggregated text, structural elements, and excerpts with citations.
Datasets
The primary dataset introduced is MegaWika 2, which contains 77 million articles across 50 languages, with 63 million scraped source texts. This is a substantial upgrade from the original MegaWika, which had 13 million articles and 30 million scraped sources. The paper also references FineWeb-Edu for URL intersection analysis and WebBrain-Raw for comparison.
Key Insights and Ideas
- Richer Structure is Key: MegaWika 2's improved article representation, capturing paragraph boundaries, sentences, and precise citation offsets, is crucial for fine-grained analysis and report generation.
- Source Quality Matters: The inclusion of ordinal text quality estimates is vital for making scraped web sources usable, as many are low-quality, irrelevant, or inaccessible (link rot).
- Multilingualism is Essential: The dataset's coverage of 50 languages highlights the growing importance of multilingual capabilities in NLP research.
- Link Rot is a Real Problem: The analysis suggests that a significant portion of cited web sources are becoming inaccessible, underscoring the need for robust archiving and verification strategies.
- LLMs for Annotation: Using LLMs for silver-standard annotation of source quality shows promise, though human oversight and careful prompt engineering are necessary.
Key Results
- Scale: MegaWika 2 is six times larger in articles and has twice as many fully scraped citations compared to MegaWika 1.
- Translation Quality: MegaWika 2 uses NLLB-200, achieving improved translation performance and significantly faster processing compared to MegaWika 1's methods.
- Source Extraction: While MegaWika 2 has more web citations, the number of sources extracted per article is lower than in MegaWika 1 for most languages, attributed partly to link rot and filtering of low-quality content.
- Source Quality Model: The developed source quality prediction model achieves good performance, with transformer-based models like XLM-RoBERTa and E5 showing strong results, particularly for distinguishing higher-quality sources.
Limitations and Future Work
- Language Coverage: While multilingual, MegaWika 2 does not cover all Wikipedia languages, and expanding this would be valuable.
- Data Normalization: The process of normalizing data across languages might obscure important linguistic or cultural variations.
- Article Quality Variation: Wikipedia's inherent quality variation and the complexities of source scraping mean that some sources may still be incorrect, irrelevant, or poorly formatted.
- Wikitext Parsing: Future work will focus on improving the parsing of structured elements like tables and infoboxes, which are currently difficult to use in their raw form.
- Source Scraping: Enhancing source download success rates and improving extraction for languages with different word separation conventions are planned.
- Enrichments: Adding article type categorization and identifying bot-written articles are potential future enrichments.
- Incremental Updates: The pipeline is designed for incremental updates, and ongoing monthly updates are planned to facilitate temporal studies.
Summary updated