Low-resource Machine Translation: what for? who for? An observational study on a dedicated Tetun language translation service

Raphaël Merx, Hanna Suominen, Adérito José Guterres Correia, Trevor Cohn

Abstract

Low-resource machine translation (MT) presents a diversity of community needs and application challenges that remain poorly understood. To complement surveys and focus groups, which tend to rely on small samples of respondents, we propose an observational study on actual usage patterns of tetun.org, a specialized MT service for the Tetun language, which is the lingua franca in Timor-Leste. Our analysis of 100,000 translation requests reveals patterns that challenge assumptions based on existing corpora. We find that users, many of them students on mobile devices, typically translate text from a high-resource language into Tetun across diverse domains including science, healthcare, and daily life. This contrasts sharply with available Tetun corpora, which are dominated by news articles covering government and social issues. Our results suggest that MT systems for institutionalized minority languages like Tetun should prioritize accuracy on domains relevant to educational contexts, in the high-resource to low-resource direction. More broadly, this study demonstrates how observational analysis can inform low-resource language technology development, by grounding research in practical community needs.

Introduction

In Natural Language Processing (NLP), "low-resource" languages—those with limited digital data—pose a significant development hurdle. Traditionally, researchers build Machine Translation (MT) systems by scraping available web data, often resulting in models biased toward news articles and religious texts. However, this approach often occurs in a vacuum, optimizing for available data rather than the practical needs of the community. This study addresses this "grounding" problem by observing how the Tetun-speaking community actually interacts with translation technology.

Methodology

The researchers analyzed 100,000 anonymized server logs from tetun.org, a popular volunteer-run MT service, collected between March and August 2024. To categorize the diverse translation requests, the team utilized Llama 3.1 8B for zero-shot domain classification, which outperformed traditional unsupervised methods like Latent Dirichlet Allocation (LDA). Additionally, they measured perplexity using the Goldfish Tetun model to evaluate how well current models handle real-world user inputs compared to standard research datasets like MADLAD-400.

Characterizing the User: The Mobile Student

Analysis of the logs reveals a specific user profile: the student power-user. Usage patterns show significant spikes in the evenings during the school week, with noticeable drops during holidays and weekends. This temporal data, combined with device metadata, suggests that the primary users are students utilizing smartphones for educational purposes.

Figure 2: Distribution of user access platforms, showing a strong preference for mobile applications, particularly via the Google Play Store.
Figure 3: Heatmap of translation requests by hour and day, indicating peak usage during weekday evenings (20:00–22:00).

Over 90% of users access the service via mobile devices (Android and iOS), with 75% specifically using a dedicated Android application. This highlights a critical need for mobile-optimized, and potentially offline, MT solutions for regions where internet connectivity may be inconsistent or expensive.

The Domain and Directionality Mismatch

The study uncovered a "sharp contrast" between what researchers provide and what users demand:

  • Directionality: While research often focuses on translating low-resource languages into high-resource ones, 70.6% of requests were for translations into Tetun (primarily from English).
  • Domain: While 83% of the MADLAD training corpus consists of news and government documents, the top user category is Science & Research (34.2%), followed by Healthcare and Education. Science-related content is 10x more prevalent in user requests than in standard training data.
  • Linguistic Flexibility: Users follow official Tetun orthography only about 71% of the time, indicating that models trained strictly on formal documents may struggle with the informal or evolving spelling used by the actual community.

Conclusion and Future Work

The performance gap is stark: the average perplexity on user-requested text was 1,774, compared to just 153 on standard corpora. This suggests current models are roughly 10 times less "familiar" with the text users actually care about.

To bridge this gap, future MT research for institutionalized minority languages should:

  1. Shift Training Focus: Prioritize the collection and curation of educational and scientific data.
  2. Develop On-Device MT: Optimize for mobile inference to serve the dominant user base.
  3. Expand Observational Studies: Apply this methodology to other minority languages to determine if the "student-on-mobile" pattern is a global phenomenon in low-resource contexts.

Summary updated

Abstract

Low-resource machine translation (MT) presents a diversity of community needs and application challenges that remain poorly understood. To complement surveys and focus groups, which tend to rely on small samples of respondents, we propose an observational study on actual usage patterns of tetun$.$org, a specialized MT service for the Tetun language, which is the lingua franca in Timor-Leste. Our analysis of 100,000 translation requests reveals patterns that challenge assumptions based on existing corpora. We find that users, many of them students on mobile devices, typically translate text from a high-resource language into Tetun across diverse domains including science, healthcare, and daily life. This contrasts sharply with available Tetun corpora, which are dominated by news articles covering government and social issues. Our results suggest that MT systems for institutionalized minority languages like Tetun should prioritize accuracy on domains relevant to educational contexts, in the high-resource to low-resource direction. More broadly, this study demonstrates how observational analysis can inform low-resource language technology development, by grounding research in practical community needs.
BibTeX
@inproceedings{merx2025low,
  author = {Raphaël Merx and Hanna Suominen and Adérito José Guterres Correia and Trevor Cohn},
  title = {{Low-resource Machine Translation: what for? who for? An observational study on a dedicated Tetun language translation service}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure