HPLT: High Performance Language Technologies

Mikko Aulamo, Nikolay Bogoychev, Shaoxiong Ji, Graeme Nail, Gema Ramírez-Sánchez, Jörg Tiedemann, Jelmer van der Linde, Jaume ZaragozaEAMT 2023 · OpenReview

Abstract

The High Performance Language Technologies (HPLT) project is a massive EU-funded initiative to democratize large-scale NLP by processing petabytes of web data into open-source datasets and models for over 100 languages. By leveraging European supercomputing infrastructure like the LUMI exascale-class system, the project aims to release not just the final models, but the entire reproducible pipeline—from raw web crawls to efficient, distilled translation and language models.

Introduction

Current state-of-the-art Large Language Models (LLMs) and Machine Translation (MT) systems face three major hurdles that limit their accessibility and scientific transparency. First is English-Centricity, where most high-performing models are optimized for English, leaving many of the world’s 7,000+ languages underserved. Second is the Data and Resource Wall; training these models requires massive datasets and specialized high-performance computing (HPC) clusters that are often proprietary or prohibitively expensive. Finally, there is a Reproducibility crisis; even when models are released, the exact data cleaning, filtering, and training pipelines are often "black boxes."

HPLT tackles these challenges by building a transparent, open-source ecosystem that bridges the gap between massive web-scale data and high-performance computing.

Methodology and Infrastructure

The project’s methodology centers on scaling existing NLP workflows to run efficiently on HPC environments. Key components include:

  • HPC Integration: Adapting NLP pipelines to run on LUMI, one of the world's fastest supercomputers. This involves making the infrastructure "NLP-aware" to handle the unique I/O and processing demands of text data at the petabyte scale.
  • Open Toolkits: The authors introduce two primary tools to manage the lifecycle of data:
  • OpusCleaner: A python-based toolkit with a web interface for downloading, examining, and filtering large-scale datasets directly on HPC clusters.
  • OpusTrainer: A large-scale data shuffler and augmenter designed to feed neural network training toolkits efficiently, supporting complex curriculum learning and stochastic data modifications.
  • Model Compression: To ensure the resulting models are usable outside of supercomputers, the project employs knowledge distillation (training smaller "student" models to mimic larger "teachers"), quantization, and other efficiency techniques.

Data Acquisition and Processing

HPLT processes staggering amounts of raw data to derive clean, usable corpora. The project utilizes 7 Petabytes (PB) from the Internet Archive and 5 PB from CommonCrawl. From this 12 PB raw crawl, the team extracts monolingual and bilingual datasets for approximately 100 languages.

The curation process adapts the ParaCrawl and MaCoCu pipelines for web-scale data processing. Once cleaned, filtered, and anonymized, these datasets are released via OPUS and LINDAT under open-source licenses, ensuring the broader research community can benefit from the massive collection effort.

Strategic Insights

Democratization through Open Pipelines

The project emphasizes that releasing the tools (like OpusCleaner) is as important as releasing the models. By making the pipelines transparent and easy to run on European HPCs, HPLT lowers the barrier for other researchers to create custom models tailored to specific needs.

Multilingualism and Data Scarcity

For languages with limited training data, the project utilizes multilingual models. This allows the model to "transfer" knowledge from high-resource languages to low-resource ones, improving performance across the board and ensuring that linguistic diversity is a core feature rather than an afterthought.

Efficiency as a Priority

Rather than just chasing the highest parameter count, HPLT focuses on "sustainable" and "reusable" models. The use of distillation and quantization suggests a focus on inference-time efficiency, making these models practical for real-world deployment on standard hardware.

Current Progress and Future Outlook

As a multi-year project initiated in late 2022, HPLT has already achieved significant milestones. The first HPLT LLMs—specifically GPT3-like models for Finnish—have been published and are currently undergoing evaluation. The project has successfully integrated its workflows with the LUMI supercomputer, paving the way for the systematic processing of the remaining raw web data.

While the computational costs of building BERT, T5, and GPT-like models for 100+ languages remain a massive undertaking, HPLT is on track to provide a free, reproducible foundation for the future of multilingual NLP. Future work will involve rigorous benchmarking using standard metrics to ensure these open models compete effectively with proprietary alternatives.

Summary updated

BibTeX
@inproceedings{aulamo2023hplt,
  author = {Mikko Aulamo and Nikolay Bogoychev and Shaoxiong Ji and Graeme Nail and Gema Ramírez-Sánchez and Jörg Tiedemann and Jelmer van der Linde and Jaume Zaragoza},
  title = {{HPLT: High Performance Language Technologies}},
  booktitle = {EAMT},
  year = {2023},
  url = {https://openreview.net/forum?id=M2-IV12D_X},
}
Figure