TL;DR
This paper introduces Wikivecs, a fully reproducible and permissively licensed dataset of dense vector embeddings for every article in Multilingual Wikipedia. By leveraging a state-of-the-art multilingual encoder and a novel dimensionality reduction technique, Wikivecs enables efficient cross-lingual analysis and the identification of content gaps across Wikipedia languages, all accessible on standard hardware.
What Problem Does This Paper Address?
Modern Natural Language Processing (NLP) heavily relies on dense vector representations (embeddings) to power applications like semantic search and retrieval-augmented generation. These embeddings are particularly valuable for multilingual tasks, as they can create a shared representational space for texts from different languages, bridging linguistic and cultural divides. Wikipedia, a vast and widely used resource for NLP research, currently lacks a comprehensive, openly licensed, and fully reproducible dataset of dense vector embeddings for all its articles. This absence hinders transparent and scalable comparative analysis across its many language editions, limiting research and development in multilingual NLP.
Methods
The core of Wikivecs is a pipeline that first embeds each Wikipedia article into a 768-dimensional vector space using Nomic-Embed-Text-v2. This model is chosen for its state-of-the-art performance and, crucially, its permissive Apache 2.0 license, which covers its training data, code, and weights, ensuring full reproducibility. To make these high-dimensional vectors more manageable, the authors apply NOMAD Projection, a scalable nonlinear dimensionality reduction algorithm, to project the embeddings into a 2-dimensional space. This 2D representation, stored in wikimap.parquet, allows for efficient visualization and analysis on standard hardware, significantly lowering the barrier to entry for researchers.
Datasets
The primary dataset used is the 2023-11-01 Wikidump dataset from Wikimedia. The output of the embedding and projection process is the Wikivecs dataset itself, which includes:
- Sharded NumPy arrays containing 768-dimensional dense vectors for each Wikipedia article.
- A
wikimap.parquetfile containing 2D positions for every article, derived from the NOMAD Projection.
The paper does not introduce new benchmarks but rather uses existing multilingual NLP benchmarks (MIRACL, MMTEB) to validate the performance of the Nomic-Embed-Text-v2 model.
Key Insights and Ideas
- Reproducibility is Key: The paper strongly emphasizes the importance of full reproducibility in NLP research, particularly for large-scale datasets. By releasing all components under permissive licenses, they aim to foster open science.
- Bridging the Gap with 2D Maps: The dimensionality reduction to 2D is a critical insight. It transforms a terabyte-scale dataset into something manageable on a laptop, democratizing access to complex multilingual data analysis.
- Identifying Content Gaps: The 2D visualization and subsequent analysis (qualitative and quantitative) effectively highlight areas where one language Wikipedia has content that is not present in another, revealing interesting linguistic and cultural disparities.
- Accessibility Drives Research: By making this resource available and usable on standard hardware, the authors aim to significantly accelerate research in multilingual NLP and content gap analysis.
Key Results
- The Wikivecs dataset provides dense vector embeddings for every article in Multilingual Wikipedia.
- The 2D projection of these embeddings allows for efficient visualization and analysis on a standard M2 Macbook Air with 24GB of RAM, completing the entire analysis in under 10 minutes.
- Qualitative analysis using the 2D map revealed content gaps, such as a cluster of Russian scientists who have articles in Russian Wikipedia but not in English Wikipedia.
- Quantitative analysis using cluster entropy identified thematic regions with low language distribution entropy, indicating potential content gaps. Manual inspection of these regions confirmed gaps related to topics like "Beetles" and "Hungarian Villages" between English and Russian Wikipedia.
Limitations and Future Work
- Scale of Full Dataset: The full 768-dimensional Wikivecs corpus is nearly a terabyte, making it challenging for many researchers to reproduce or analyze in its entirety. The 2D projection mitigates this, but generalizing findings from 2D to 768D is an inherent limitation.
- Manual Verification: The process of verifying content gaps was highly manual, limiting the scope of the analysis to one language pair (English-Russian) and small evaluation sample sizes.
- Future Work: The authors hope their dataset will accelerate community efforts to understand and validate content gaps across more language pairs. They also suggest expanding the quantitative and qualitative analyses.
Summary updated