WikiOmnia: filtration and evaluation of the generated QA corpus on the whole Russian Wikipedia

Dina Pisarevskaya, Tatiana Shavrina

TL;DR

WikiOmnia is a massive, automatically generated Question Answering (QA) dataset covering the entirety of Russian Wikipedia. By combining large generative models (ruT5 and ruGPT-3) with a rigorous multi-stage filtration pipeline, the authors produced over 3.5 million high-quality QA pairs—a significant leap for Russian NLP resources.

What Problem Does This Paper Address?

In the world of Natural Language Processing, Question Answering (QA) is a foundational task. However, creating high-quality datasets like the Stanford Question Answering Dataset (SQuAD) usually requires expensive, slow manual annotation by humans. This "human bottleneck" limits the size and diversity of available data.

For the Russian language, the primary resource (SberQuAD) contains only about 50,000 pairs. To train more robust, "common sense" AI systems, we need millions of examples. The authors address this by asking: Can we use AI to build a dataset for AI? Specifically, they look at how to generate millions of QA pairs from Wikipedia summaries and, more importantly, how to filter out the "hallucinations" and nonsense that generative models often produce.

Methods

The authors developed a fully automated pipeline consisting of two main phases: Generation and Filtration.

1. Generation

They fine-tuned two state-of-the-art Russian language models on SberQuAD to learn the format of "Paragraph $\rightarrow$ Question $\rightarrow$ Answer":

  • ruGPT-3 XL: A large autoregressive model.
  • ruT5-large: An encoder-decoder model. These models were tasked with generating three QA pairs for every article summary in the Russian Wikipedia (approx. 2.68 million articles).

2. Filtration (The "Secret Sauce")

To ensure the synthetic data was actually useful, they applied a "rich heuristic" approach:

  • Interrogative Check: Dropped questions with more than one interrogative pronoun (e.g., "Who did what where?").
  • The BERT "Gold" Standard: They used a separate, pre-trained Russian BERT model to answer the generated questions. If the generative model's answer didn't match the BERT model's answer (using a 70% lemma overlap threshold), the pair was discarded.
  • NER Verification: They used Named Entity Recognition (NER) to ensure that any names, places, or dates mentioned in the question or answer actually appeared in the source Wikipedia text.
  • Similarity Ratios: Used Levenshtein distance to remove near-duplicate questions.

Datasets

  • WikiOmnia (New): The primary contribution. It contains a "raw" set of 15.9 million triplets and a "filtered" set of 3.5 million verified triplets.
  • SberQuAD: Used as the baseline and for fine-tuning the generative models.
  • Transfer Domains: To test the pipeline's versatility, they applied it to news (Gazeta), social media (Taiga), product reviews, and classical Russian literature.

Key Insights and Ideas

  • ruT5 > ruGPT-3: One of the most striking findings was the performance gap between the two architectures. While ruGPT-3 is famous for its generative prowess, ruT5 was far more reliable for QA. After filtration, over 3.4 million ruT5 pairs remained, compared to only 160,000 for ruGPT-3.
  • Synthetic Data as a "Warm-up": The authors found that training a model on WikiOmnia before fine-tuning it on human-annotated SberQuAD data led to better performance than training on human data alone. This suggests synthetic data is excellent for "pre-conditioning" models on the mechanics of QA.
  • Topic Sensitivity: The pipeline performed best on sports articles (which have simple, factual structures) and struggled with social media (which is opinionated and lacks clear factual anchors).
  • Diversity vs. Quality: While the filtration process significantly reduced the number of samples, it actually increased the linguistic diversity (measured via Self-BLEU) of the remaining questions for ruT5.

Key Results

  • Scale: The project successfully processed the entire Russian Wikipedia, resulting in the largest QA resource for the language to date.
  • Performance Boost: In "Experiment Set 2," a model fine-tuned on 100,000 WikiOmnia samples followed by SberQuAD achieved an F1 score of 86.26, outperforming the baseline model trained only on SberQuAD (85.92).
  • Cross-Domain Success: The pipeline proved robust enough to generate over 1,500 high-quality pairs from product reviews and 3,200 from news texts with zero manual intervention.

Limitations and Future Work

  • Numerical Errors: The filters still occasionally miss errors involving numbers (e.g., getting a year wrong if the BERT model and the generative model hallucinate the same wrong date).
  • Hallucinations: A tiny fraction (0.008%) of ruGPT-3 questions still contained information not present in the source summary.
  • Future Directions: The authors suggest incorporating more sophisticated paraphrasing checks and developing specific filters for numerical and temporal logic to further clean the "raw" 15.9 million triplets.

Summary updated

Abstract

The General QA field has been developing the methodology referencing the Stanford Question answering dataset (SQuAD) as the significant benchmark. Compiling factual questions datasets requires manual annotations, limiting the training data’s potential size. We present the WikiOmnia dataset, a new publicly available set of QA pairs and corresponding Russian Wikipedia article summary sections, composed with a fully automated generation and filtration pipeline. To ensure high quality of generated QA pairs, diverse manual and automated evaluation techniques were applied. The WikiOmnia pipeline is available open-source and is also tested for creating SQuAD-formatted QA on other domains, like news texts, fiction, and social media. The resulting dataset includes two parts: raw data on the whole Russian Wikipedia (7,930,873 QA pairs with paragraphs for ruGPT-3 XL and 7,991,040 QA pairs with paragraphs for ruT5-large) and cleaned data with strict automatic verification (over 160,000 QA pairs with paragraphs for ruGPT-3 XL and over 3,400,000 QA pairs with paragraphs for ruT5-large).
BibTeX
@misc{pisarevskaya2022wikiomnia,
  author = {Dina Pisarevskaya and Tatiana Shavrina},
  title = {{WikiOmnia: filtration and evaluation of the generated QA corpus on the whole Russian Wikipedia}},
  year = {2022},
  eprint = {2204.08009},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.08009},
}
Figure