In natural language processing, "low-resource" is often treated as a synonym for "small dataset." To study these scenarios, researchers frequently take a massive, clean dataset—like the English Wikipedia—and artificially shrink it to a few thousand sentences. This practice, known as downsampling, creates a controlled environment to test how algorithms behave when data is scarce.
However, a growing body of evidence suggests that these simulations are fundamentally flawed. By treating low-resource languages as merely "smaller versions" of high-resource ones, we fall into a proxy fallacy. As David Grangier, Natalie Schluter, and their colleagues argue in High-Resource Methodological Bias in Low-Resource Investigations, downsampled data possesses statistical properties that are radically different from those of naturally scarce languages.
The Structural Mismatch
The first reason simulations fail is linguistic. High-resource languages (HRLs) like English are often morphologically "simple" compared to many low-resource languages (LRLs). When you downsample English, you still have a language with relatively few word forms per root.
In contrast, many LRLs are morphologically rich or polysynthetic. In Tackling the Low-resource Challenge for Canonical Segmentation, Manuel Mager and Katharina Kann highlight that models performing well on simulated data often struggle with the high morpheme-to-word ratios of authentic LRLs. Similarly, P. Godard and Laurent Besacier found that unsupervised word discovery algorithms see a significant performance drop when moving from simulated data to Mboshi, a Bantu language, due to phonetic and tonal complexities that English-based simulations simply don't have Preliminary Experiments on Unsupervised Word Discovery in Mboshi.
Even at the same token count, an HRL and an LRL do not have the same "vocabulary richness." A small slice of a large, diverse corpus retains a higher unique word count than a naturally small corpus, leading to a systemic mismatch in data complexity that inflates model performance in simulations.
The Noise of the Real World
Simulated datasets are usually derived from highly curated sources like news or Wikipedia. Authentic low-resource data, however, is "messy." It often comes from field linguistics notes, unwritten speech, or social media with non-standardized scripts.
Gina‐Anne Levow and Emily M. Bender emphasize in the STREAMLInED Challenges that machine learning models developed on clean simulations fail to account for the specific noise and structural irregularities found in endangered language documentation. This "noise gap" means that strategies like Active Learning, which look promising on downsampled datasets, may become unstable when faced with the extreme noise of genuine scarcity COMET-QE and Active Learning for Low-Resource Machine Translation.
The Mirage of Performance
The most dangerous aspect of the proxy fallacy is that it creates a "mirage" of success. When we evaluate a model on a downsampled dataset, we often benefit from hidden advantages:
- Lemma Overlap: In (Un)solving Morphological Inflection, Omer Goldman and Reut Tsarfaty show that high accuracy scores often stem from the model "memorizing" roots it saw during training. In true low-resource settings, the model is much more likely to encounter entirely new vocabulary, causing performance to plummet.
- Auxiliary Resources: Simulations often assume the existence of "hidden" resources. For example, Shuyan Zhou and Graham Neubig point out that many "low-resource" benchmarks still rely on bilingual entity maps or Wikipedia coverage that simply do not exist for languages like Oromo or Tigrinya Towards Zero-resource Cross-lingual Entity Linking.
- Pre-training Bias: Transfer learning is often touted as the solution for LRLs. However, Jeremy Barnes and colleagues note that the perceived effectiveness of models like BERT is overstated for languages where massive pre-training corpora are unavailable Hierarchical models vs. transfer learning for document-level sentiment classification.
Redefining Scarcity
To move past the proxy fallacy, the field is shifting toward a multi-dimensional definition of "low-resource." As Hellina Hailu Nigatu and Monojit Choudhury argue in The Zeno's Paradox of 'Low-Resource' Languages, progress should be measured not just by data volume, but by a language's socio-political context, the availability of human experts, and the presence of digital infrastructure.
By prioritizing authentic data—even when it is messy and difficult to work with—researchers can ensure that the tools they build actually serve the communities they are intended for, rather than just solving an artificial puzzle created by downsampling English.