TL;DR
This paper presents an observational study of actual user behavior on tetun.org, a dedicated machine translation (MT) service for the Tetun language. By analyzing 100,000 translation requests, the study reveals that users, primarily students, translate educational content from high-resource languages into Tetun, a pattern that contrasts sharply with existing Tetun corpora. This highlights a critical need for MT systems to prioritize accuracy in specific domains and directions relevant to low-resource language communities.
What Problem Does This Paper Address?
Machine translation (MT) for low-resource languages faces a significant challenge: a disconnect between the available training data and the actual needs of users. Existing research often relies on surveys or focus groups with small sample sizes, which may not fully capture real-world usage patterns. Furthermore, low-resource language corpora are frequently dominated by specific domains (e.g., news, religious texts), leading to MT systems that lack robustness across diverse applications. This paper addresses this gap by conducting an observational study of a dedicated MT service for Tetun, a low-resource language spoken in Timor-Leste, to understand who uses it, what they translate, and how their usage patterns align with or diverge from existing linguistic resources.
Methods
The researchers employed an observational approach, analyzing server logs from tetun.org, a widely-used MT service for the Tetun language. This involved:
- Data Collection: A sample of 100,000 translation requests from March to August 2024, including timestamps, input text, MT output, source/target language codes, and device operating system.
- User Profiling: Analyzing monthly active user data from Google Analytics and app stores to understand user demographics and device preferences.
- Domain Classification: Developing a methodology to classify translation inputs by topic, provenance (source of text), and genre. This involved using Latent Dirichlet Allocation (LDA) for topic identification and then leveraging pre-trained language models (LLMs) like Llama 3.1 8B for classification on the high-resource side of the language pair.
- Comparison with Corpora: Comparing the identified domain characteristics of MT inputs with those of the MADLAD-400 Tetun corpus to highlight discrepancies.
- Perplexity Analysis: Measuring the perplexity of MT inputs using a monolingual Tetun language model (Goldfish Tetun) to assess how well these inputs are represented in existing corpora and their potential impact on MT performance.
Datasets
- tetun.org Server Logs: A dataset of 100,000 translation requests collected between March and August 2024. This is the primary dataset for analyzing user behavior and MT inputs.
- Google Analytics & App Store Data: Used to gather summary monthly active user data by country and device.
- MADLAD-400 Tetun Corpus: The largest available Tetun corpus (40,367 documents) used as a benchmark for comparing the domain distribution of MT inputs against existing linguistic resources. The Tetun portion of this corpus was translated to English for analysis by models that do not support Tetun.
- Labadain Tetun Corpus: The second-largest Tetun corpus, used for perplexity measurements, excluding sentences also present in MADLAD.
Key Insights and Ideas
- User-Centricity is Crucial: The study strongly advocates for grounding low-resource MT development in actual user behavior, moving beyond assumptions based on available corpora.
- Students as Primary Users: The findings suggest that a significant portion of tetun.org users are students, indicating a strong demand for MT in educational contexts.
- Domain Mismatch is a Major Hurdle: The discrepancy between the domains users translate (education, science, health) and the domains prevalent in existing Tetun corpora (news, government) is a key challenge for MT performance.
- Directionality Matters: The high-resource to low-resource direction (e.g., English to Tetun) is significantly more in demand than the reverse, highlighting the need for MT systems to excel in facilitating access to information into the low-resource language.
- Mobile-First Approach: The prevalence of mobile device usage for MT necessitates a focus on efficient, on-device MT solutions.
Key Results
- User Base: Approximately 93.4% of Android monthly active users and 77% of website monthly active users are located in Timor-Leste, suggesting a predominantly local user base.
- Device Usage: Mobile devices, particularly Android, are the primary means of accessing tetun.org, accounting for a large majority of users.
- Translation Direction: Tetun is the target language in over 70% of requests, with English to Tetun being the most frequent pair (46.7%).
- Text Length: Shorter text inputs are most common, with a median of 8 words, but a long tail exists for longer documents.
- Topic Demand: "Science & research" (34.2%), "Healthcare & medicine" (23.9%), and "Education" (22.8%) are the most in-demand topics for MT inputs. This contrasts sharply with the MADLAD corpus, where "Government & socio-economic issues" (83.1%) and "Law & regulations" (28.2%) are dominant.
- Provenance: "Education & research material" constitutes over half (55.9%) of MT inputs, while "News articles & press releases" dominate the MADLAD corpus (61.2%).
- Genre: Official Tetun spelling is followed loosely in MT inputs (median 71% correct), with higher variance than in the MADLAD corpus.
- Perplexity: MT inputs exhibit significantly higher perplexity (average 1,774) compared to a Tetun corpus (Labadain, average 153), indicating that user-generated content is less represented in existing corpora, potentially leading to lower MT performance.
Limitations and Future Work
- System Design Impact: The tetun.org service includes a dictionary and spell checker, which might influence user behavior and the number of users who solely use the MT functionality. The limited language pairs supported (English, Portuguese, Indonesian) also affect usage patterns.
- Location Data Accuracy: User location data might not perfectly reflect their actual location or language background due to VPN usage or devices registered abroad.
- Data Bias: Server logs only capture successful translation attempts, potentially biasing the data towards domains or language pairs where tetun.org performs better.
- Generalizability: Findings are specific to Tetun and may not directly apply to all low-resource languages, especially oral or endangered ones.
Future work should focus on:
- Domain Coverage: Prioritizing MT development for domains not well-covered by existing corpora, particularly health and science education.
- Directionality: Emphasizing accuracy in the high-resource to low-resource direction.
- Short Texts: Improving MT performance on very short texts.
- Mobile MT: Developing open-source, on-device MT solutions for mobile devices.
- HCI Research: Focusing on MT perception and improvement for students in low-resource contexts.
Summary updated