The Cost of Training NLP Models: A Concise Overview

Or Sharir, Barak Peleg, Yoav ShohamarXiv 2020 · arXiv

Introduction

In the current "arms race" of Natural Language Processing (NLP), the research community has shifted toward a mega-scale, brute-force statistical approach. While this has yielded unprecedented performance, it has also erected a massive financial barrier to entry. As models scale to billions of parameters, the cost of training has skyrocketed, with some projects now carrying a "list price" tag exceeding $10 million. This paper analyzes the primary drivers of these costs—data size, model complexity, and training volume—while exploring the economic disparity between NLP and other fields like computer vision.

The Economic Drivers of Scale

The cost of training a state-of-the-art (SOTA) model is determined by three primary dimensions: data size, model size, and training volume.

Model Size and Complexity

Model size, measured in billions of parameters, has seen exponential growth. For instance, while the original BERT-Large utilized approximately 340 million parameters (0.3B), subsequent models like T5 and Turing-NLG have pushed these boundaries to 11 billion and 17 billion parameters, respectively. This increase in "width" and "depth" directly translates to higher computational requirements.

Growth in model size (billions of parameters) across major NLP architectures.

Training Volume and Data

Training volume represents the total number of tokens processed during pre-training—a product of sequence length, training steps, and batch size. Modern models now routinely process trillions of tokens, a massive leap from earlier baselines.

Training volume in trillion tokens for various SOTA models.

The datasets fueling this growth have also expanded. From the 15 GB Wikipedia and BookCorpus used for BERT, the field has moved to the Colossal Clean Crawled Corpus (C4), a 35-billion-word dataset used for Google’s T5 model. Unlike computer vision, where supervised learning on labeled data is common, NLP has shifted toward purely self-supervised learning on these massive datasets, requiring significantly longer training cycles to extract useful signals from raw text.

NLP vs. Computer Vision: An Architectural Tax

A key insight is that NLP is inherently more expensive to scale than computer vision (CV). Images possess "locality" and "smoothness," where a pixel's value is highly related to its neighbors, allowing for efficient Convolutional Neural Networks (CNNs). In contrast, language is "compositional" and "coupled." Words far apart in a sentence can drastically alter each other's meaning, necessitating the Transformer architecture. While powerful, Transformers are computationally expensive, particularly as sequence lengths and model depths increase.

The "Fully-Loaded" Cost Reality

The "list price" of a single successful training run is often a misleading metric. In practice, researchers must account for a "fully-loaded" multiplier. Achieving SOTA results typically requires 10–20 runs to account for hyperparameter tuning, debugging, and the stochastic nature of training.

Estimated project costs (at cloud list prices) illustrate this burden:

  • BERT-Large (340M parameters): $10k – $200k for a fully-loaded project.
  • GPT-2 (1.5B parameters): $80k – $1.6M for a fully-loaded project.
  • T5 (11B parameters): A single run is estimated at $1.3 million, with the total project cost likely hitting $10 million.

Conclusion and Future Directions

The current trajectory of adding more "brawn" (compute) is increasingly unsustainable. While hardware optimizations and cloud competition may lower raw compute costs, the field requires a shift toward "brains"—incorporating structured knowledge and more efficient architectures. Emerging models like Reformer or ALBERT, which utilize weight-sharing or heuristics to reduce complexity, represent a necessary move toward democratizing NLP research and reducing its massive economic footprint.

Summary updated

Abstract

We review the cost of training large-scale language models, and the drivers of these costs. The intended audience includes engineers and scientists budgeting their model-training experiments, as well as non-practitioners trying to make sense of the economics of modern-day Natural Language Processing (NLP).
BibTeX
@misc{sharir2020cost,
  author = {Or Sharir and Barak Peleg and Yoav Shoham},
  title = {{The Cost of Training NLP Models: A Concise Overview}},
  year = {2020},
  eprint = {2004.08900},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2004.08900},
}
Figure