Masked language models learn by hiding a word and being scored on whether they guess the exact string that was there. For dialects that have no standard spelling, such as Swiss German or colloquial Arabic, this scoring is often unfair. The same word appears as "chuchichäschtli", "chuchikästli", or a regional synonym, and the model is penalized for predicting a form that a native speaker would accept. This proposal uses a game that dialect speakers already enjoy playing to supply two things a web scraper cannot. The first is new dialect text aimed at exactly the sentences where the current model is most confused. The second is player-validated judgments that two surface forms mean the same thing. Both feed the model's continued pre-training, meaning further training of an existing base model on domain text before any task-specific tuning. The equivalence judgments then change the training objective so that valid variants stop counting as errors.
The opportunity is that gamified collection and domain pre-training have so far stayed apart. Dia-Lingle runs a dialect-rewriting game whose item difficulty comes from a small classifier's uncertainty. It never trains a transformer on what it collects. Dialect pre-training work such as AlcLaM, meanwhile, treats its corpus as a fixed scrape of social-media comments. In plain terms, the claim is this: for unstandardized dialects, a small amount of game data is worth more per token than scraped text. The reason is not volume. It is that the game targets the model's blind spots and tells the training objective which "wrong" answers are actually right.
Core Claim
Continued pre-training of a dialect encoder improves more when two conditions hold together. The game items must be chosen by the encoder's own masked-prediction loss, and the masked-word objective must be corrected with equivalence sets that players validated. Either ingredient alone should help less. The gains should show up most clearly in robustness to spelling and lexical variation and in fine-grained dialect identification. We do not expect broad general-language gains, because the game will produce on the order of tens of thousands of sentences, not the tens of millions of tokens AlcLaM scraped.
The objective change comes from Language Model Pre-training on True Negatives. That paper found that over 7% of masked-prediction instances in English penalize valid synonyms. It corrected them by masking the gradient for synonyms found in a thesaurus lookup. Its correction cut the score drop under adversarial synonym swaps on SQuAD from 6.0 to 2.66 points. Dialects have no thesaurus, so that is where the game comes in. We replace the lookup with a player-built equivalence set $E(y)$ for each masked word $y$, and use a set-level likelihood:
$$\mathcal{L} = -\log \sum_{v \in E(y)} p_\theta(v \mid x_{\setminus y})$$
Here $x_{\setminus y}$ is the sentence with $y$ masked, and $p_\theta(v \mid \cdot)$ is the model's probability of filling the gap with variant $v$. For multi-piece words, $p_\theta$ is the product of the word's subword probabilities under whole-word masking. When $E(y)=\{y\}$, this reduces to ordinary masked language modeling. True Negatives' hard correction corresponds to dropping the penalty on any $v \in E(y)$. We test both forms.
Research Program
The collection side extends Dia-Lingle's Swiss German setup, which already has a 2,760-group parallel corpus and a reported usability score of 77.78. We keep its Quiz mode, in which players rewrite a High German sentence into their dialect. We change how items are selected. Dia-Lingle chose items by the entropy of a small fastText classifier. We instead rank candidate source sentences by the current encoder's average masked-word loss on the dialect variants collected so far for each region. Sentences that have no variant yet for a region get a bonus.
We also add an agreement round that follows the output-agreement design of CapCap. Two players who cannot communicate see an aligned pair of words drawn from different rewrites of the same sentence and each answer whether the two mean the same thing. A pair enters $E(y)$ only when independent players agree and the pair's players also pass hidden gold items. This guards against the collusion toward trivial consensus that output-agreement games are known to invite. The design choice to make agreement the condition for progressing, rather than a side chore, follows Vannella et al.. They found game-embedded validation beat paid crowdsourcing on agreement with experts (82.5% vs 66.2% on concept-image pairs) at zero marginal cost, and that a non-game wrapper attracted no voluntary players.
The training side runs three collection rounds. After each round we continue pre-training the encoder on the accumulated game text mixed with existing Swiss German text, recompute losses, and reselect items. We keep the domain data isolated rather than diluting it into a large general mix. Durrani et al. found that mixing small high-value corpora into large generic ones drowns the domain signal. As a secondary arm, we may also tag each training sentence with its provenance, meaning the game mode and the agreement score. This follows the metadata-prefix recipe of MeCo, which prepends source tags and removes them for a final cooldown. That arm is exploratory and is not part of the core claim.
To test portability, we repeat a single round on Gulf and Levantine Arabic. The starting point is AlcLaM, a BERT-base encoder trained from scratch on 13GB of scraped dialect comments. That run asks whether game data adds anything on top of a model that already used scraping well.
Baselines
The naive reference point is the base encoder with no continued pre-training: a German or multilingual BERT-style model for Swiss German, and AlcLaM for Arabic. The main scraping-style competitor is the same encoder continued-pre-trained on an equal number of tokens of existing Swiss German text, drawn from SwissDial and Dia-Lingle's released corpus, with standard masked prediction. This is the strongest competitor because it is what AlcLaM shows works. To beat it, the full system must improve over it on variant robustness and dialect identification at matched tokens, steps, and learning-rate sweep budget. For Swiss German we found no reported result on Lacuna for a pre-trained transformer on Dia-Lingle's Swiss German split or on SwissDial dialect identification; Dia-Lingle reports fastText F1 only for other languages (0.446 Japanese to 0.973 Korean). For Arabic, the reference is the best reported city-level system in the MADAR Shared Task, with macro-F1 of 67.32% on Subtask 1, where deep models had struggled to beat n-gram baselines. AlcLaM's 99.7% on the coarser MADAR-2 task is near ceiling and unsuitable as a target.
Three ablations isolate the ingredients. The first uses game data collected with random item selection at equal volume, which isolates loss-targeting. The second uses targeted game data with ordinary masked prediction, which isolates the equivalence correction. The third applies the correction with equivalence sets induced automatically from SwissDial's eight-dialect parallel alignments, using edit distance and embedding neighbors, which isolates the human judgment. All arms share data budget, compute, and tuning budget. Three seeds per arm are reported with confidence intervals.
Why not just…?
Why not just scrape more dialect text? AlcLaM shows scraping can work when a dialect is widely written online. Dia-Lingle reports that many Swiss and minority dialects are mostly spoken, so scraping yields little. MADAR's city-level results show sparsity hurting even for Arabic. Scraping is still the main threat for Arabic, which is why the Arabic arm starts from AlcLaM. If game data adds nothing there, we will report that the method's value is confined to dialects that are poorly written online.
Why not just generate dialect rewrites with a large language model? DoPAMine found synthetic domain text unreliable enough that it used generated text only as search seeds, mining real human text for training. That finding concerned medical and finance text, not dialects. We therefore add a matched-token arm of LLM-generated Swiss German rewrites, with and without automatically induced equivalences. If this arm matches the game, the game's value reduces to equivalence judgments alone, and the third ablation will show whether those survive.
Why not just derive equivalences offline from parallel corpora? SwissDial already aligns eight dialects with High German, so equivalences could be mined without any players. This is the most serious alternative, and the third ablation tests it directly. Human validation should matter most where alignment is ambiguous, such as for regional synonyms that differ entirely in form. If automatic sets perform equally well, the honest conclusion is that the game is a text-collection tool and not an objective-shaping one.
Why not just fine-tune on the game data instead of pre-training on it? Männistö found domain pre-training a stronger predictor of task performance than the fine-tuning method used. MediTOD shows fine-tuned models degrading out of domain (NLU F1 from 0.7139 to about 0.52). Both results come from other domains, so we test the question directly. One arm puts the same game sentences into task fine-tuning rather than continued pre-training, and we evaluate it on dialects held out of the game.
Proposed Experiments
We need test sets that the game did not produce, so evaluation uses held-out SwissDial sentences and a held-out Dia-Lingle split. The first evaluation is eight-way Swiss German dialect identification, measured in macro-F1. The second is cross-dialect sentence retrieval: given a sentence in one dialect, find its translation in another dialect among SwissDial's parallel sentences, measured by top-1 accuracy. The third is a variant-swap robustness test modeled on True Negatives' SQuAD synonym-swap test. We replace words in test sentences with held-out, expert-checked alternative spellings and measure how far each model's dialect-identification and retrieval scores fall. For Arabic, the evaluation is MADAR Subtask 1 city-level macro-F1.
The decisive checks follow from the ablations. If random-selection game data matches loss-targeted data, the claim that targeting matters fails. If ordinary masked prediction matches the corrected objective on the variant-swap test, the claim that equivalence correction matters fails. If automatic equivalence sets match player-validated ones, the human-signal claim fails. The full proposal survives only if the complete system beats matched-token scraping on variant robustness and fine-grained identification, and each ablation loses something. We will also log per-round player retention, because Dia-Lingle flags early misclassifications as a trust hazard and loss-targeted items may be harder. If retention collapses, the item-selection rule needs a difficulty cap before the loop can scale.
References
- Dia-Lingle: A Gamified Interface for Dialectal Data Collection
- AlclaM: Arabic Dialect Language Model
- Language Model Pre-training on True Negatives
- Capcap: an output-agreement game for video captioning
- Validating and Extending Semantic Knowledge Bases using Video Games with a Purpose
- A Deep Fusion Model for Domain Adaptation in Phrase-based MT
- Metadata Conditioning Accelerates Language Model Pre-training
- SwissDial: Parallel Multidialectal Corpus of Spoken Swiss German
- The MADAR Shared Task on Arabic Fine-Grained Dialect Identification
- DoPAMine: Domain-specific Pre-training Adaptation from seed-guided data Mining
- A Comparative Study of PEFT Methods for Python Code Generation
- MediTOD: An English Dialogue Dataset for Medical History Taking with Comprehensive Annotations