GrEmLIn: A Repository of Green Baseline Embeddings for 87 Low-Resource Languages Injected with Multilingual Graph Knowledge

Daniil Gurgurov, Rishu Kumar, Simon OstermannNAACL 2025 · DOI

Abstract

GrEmLIn introduces a "green" repository of static word embeddings for 87 low-resource languages (LRLs), offering a high-efficiency alternative to power-hungry Large Language Models (LLMs). By injecting multilingual knowledge from ConceptNet into traditional GloVe vectors via a parameter-free approach, GrEmLIn provides representations that outperform state-of-the-art contextual models in lexical similarity and remain surprisingly competitive on downstream tasks like sentiment analysis. This work addresses the "curse of multilinguality" and the prohibitive computational costs of modern NLP for resource-constrained environments.

Introduction

While contextual embeddings (e.g., BERT, GPT) have revolutionized NLP, they impose a significant computational "tax." Training and inference require substantial hardware resources, which are often unavailable in low-resource scenarios or on edge devices. Furthermore, these models frequently struggle with low-resource languages due to the lack of massive training corpora. Existing static embedding repositories, while efficient, are often outdated or limited in language coverage.

GrEmLIn bridges this gap by providing a centralized, high-quality repository of "green" baseline embeddings specifically tailored for 87 mid- and low-resource languages. These embeddings are designed to be both environmentally sustainable and functionally robust.

Methodology

The core innovation in GrEmLIn is a novel method for merging distributional information from text with relational information from knowledge graphs:

  1. GloVe Training: Standard GloVe embeddings are trained on the CC100 dataset for each of the 87 languages.
  2. Graph Embedding (PPMI): Structured relationships (synonyms, antonyms, etc.) are extracted from ConceptNet 5.7 using Pointwise Mutual Information (PPMI).
  3. Shared Space via SVD: For words present in both GloVe and ConceptNet, vectors are concatenated and reduced via Singular Value Decomposition (SVD) to create a shared, dense latent space.
  4. Linear Transformation: To benefit the entire vocabulary, a linear projection matrix is learned to map all original GloVe vectors into the new, graph-enhanced space.

This approach is "parameter-free" at inference time; the final result is a simple lookup table, requiring no neural network forward pass.

Experimental Setup

The embeddings were evaluated across several benchmarks:

  • Intrinsic Evaluation: MultiSimLex for measuring semantic similarity between word pairs.
  • Extrinsic Evaluation:
    • Sentiment Analysis (SA): Data for 23 languages.
  • Topic Classification (TC): The SIB-200 dataset.
  • Natural Language Inference (NLI): The XNLI dataset.

Key Results and Discussion

Lexical Similarity

GrEmLIn embeddings significantly outperformed state-of-the-art contextual models like E5-base and XLM-R on the MultiSimLex task. As shown in the results, the injection of structured knowledge provides a vital semantic anchor that raw text co-occurrence often misses in low-resource settings.

Comparison of GrEmLIn (GP) vs. Contextualized models on Lexical Similarity. GrEmLIn variants (GP(S) and GP(A)) significantly outperform E5-base in this intrinsic task.

Downstream Performance and Efficiency

On tasks like Sentiment Analysis and NLI, the performance gap between GrEmLIn's static embeddings and top-tier contextual models was remarkably small—often between 5-10%. This suggests that for many practical applications, the massive energy consumption of LLMs may not yield proportional benefits.

Performance on downstream tasks (e.g., Sentiment Analysis). GrEmLIn remains competitive with contextual models, showing a narrow performance gap despite having zero inference-time parameters.

However, contextual models maintained a clear lead (~20% gap) in Topic Classification, likely because this task relies more heavily on document-level context that simple word-summing cannot replicate.

Conclusion

GrEmLIn demonstrates that static embeddings, when enhanced with multilingual graph knowledge, remain a powerful and sustainable tool for NLP. By providing a resource footprint orders of magnitude smaller than contextual models, GrEmLIn enables high-quality language technology for the "next billion users" without the associated environmental or financial costs of LLMs. Future work will explore more sophisticated fusion techniques, such as Graph Neural Networks, while maintaining the efficiency of static outputs.

Summary updated

BibTeX
@inproceedings{gurgurov2025gremlin,
  author = {Daniil Gurgurov and Rishu Kumar and Simon Ostermann},
  title = {{GrEmLIn: A Repository of Green Baseline Embeddings for 87 Low-Resource Languages Injected with Multilingual Graph Knowledge}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.findings-naacl.67},
}
Figure