Measuring What Counts as a Correct Spelling in Unwritten Dialects

Many spoken dialects have no agreed way of being written. In Swiss German, roughly half of all word types in a corpus of text messages appear in at least two spellings, and a word as common as nächste ("next") appears in more than 25 (Crowdsourcing Swiss Dialect Transcriptions for Assessing Factors in Writing Variations). Speech recognizers for such dialects are still scored by word error rate (WER), which counts a word as wrong unless it matches one reference spelling character for character. The result is that an equally valid spelling counts as an error, and nobody can say how much of a reported WER is real misrecognition. This note proposes collecting the missing measurement cheaply. Several native speakers transcribe the same short clip on their phones, a model offers candidate spellings to speed them up, and a chance-corrected agreement statistic is used twice: once on which word was heard, and once on how it was spelled. Words where speakers agree on identity but differ on form are legitimate spelling variants. Words where they disagree on identity are genuine hearing ambiguity or error.

The two-sentence version is this. Have several native speakers transcribe each clip on their phones and measure their agreement separately at the level of word identity and at the level of spelling. The gap between the two defines a per-word set of accepted spellings that a recognizer can be scored against. The catch, and the main scientific question, is that model suggestions shown to transcribers may themselves erase the spelling variation we are trying to measure. The study therefore randomizes whether and how suggestions are shown and measures that distortion directly.

Core Claim

Transcription agreement in a non-standardized dialect has two layers that current practice merges. Let $\alpha_{\text{lex}}$ be Krippendorff's alpha computed over aligned word slots after mapping each spelling to a lexical identity, and let $\alpha_{\text{surf}}$ be alpha over the same slots using normalized character edit distance between spellings as the disagreement function. Krippendorff's alpha is a chance-corrected agreement coefficient that tolerates missing raters and arbitrary distance functions; 1 means perfect agreement and 0 means chance. We define the orthographic dispersion of a clip as

$$D = \alpha_{\text{lex}} - \alpha_{\text{surf}},$$

which is large when speakers hear the same words but write them differently. We claim three things. First, $D$ is substantial and stable for Swiss German lay transcription. Second, slots with high lexical agreement yield a variant set: every spelling produced independently by at least two transcribers. Scoring recognizers against these sets ranks them in closer accord with native-speaker acceptability judgments than single-reference WER does. Third, model-suggested spellings on the phone raise speed but also raise $\alpha_{\text{surf}}$ artificially through anchoring. Offering several diverse candidates rather than one recovers most of the speed while keeping spelling diversity close to the unprimed level.

Relation to Prior Work

The motivating evidence comes from two papers. Crowdsourcing Swiss Dialect Transcriptions for Assessing Factors in Writing Variations measured spelling entropy in static text (SMS4science, ArchiMob) and proposed having laypeople transcribe 2–5 second audio chunks in their own spelling. It reports no agreement statistic between transcribers and uses no model assistance. We adopt its clip format and character alignment method and add the two-layer agreement analysis and the assistance experiment. Estimating the Level of Dialectness Predicts Inter-annotator Agreement in Multi-dialect Arabic Datasets showed across 15 Arabic text-classification datasets that agreement falls as text becomes more dialectal when annotators do not share the variety. That work concerns classification labels on existing text. We test the analogous prediction for transcription of speech, using whether the transcriber's and speaker's dialect regions match.

The agreement methodology is borrowed. Superlim: A Swedish Language Understanding Evaluation Benchmark treats a model as one more annotator and scores it with Krippendorff's alpha against gold labels on standardized Swedish. We do the same for a speech recognizer, scoring its output as an additional rater against the human panel, but with a distance function that encodes spelling dispersion.

Two papers constrain the design and serve as competitors. STT4SG-350: A Speech Corpus for All Swiss German Dialect Regions avoids dialect writing entirely by recording oral translations of Standard German prompts and scoring against Standard German. That gives a clean metric but cannot evaluate or produce written dialect, which is how Swiss speakers actually text. Modeling Dialectal Variation for Swiss German Automatic Speech Recognition generates spelling variants offline with a phoneme-to-grapheme model seeded from about 11,000 expert words. A phoneme-to-grapheme model maps a pronunciation to plausible written forms. That work maps variants back to normalized forms and reports a 9% relative WER gain. We reuse this generator as the suggestion engine, but its variants are never validated by speakers, which is what our panel supplies.

The human-in-the-loop literature supplies the speed argument and the main warning. Building a Role Specified Open-Domain Dialogue System Leveraging Large-Scale Language Models found that reviewing model output was 13.3 times faster than writing from scratch. Human-Machine Collaboration Approaches to Build a Dialogue Dataset for Hate Speech Countering measured post-editing effort with HTER, the fraction of edits a human makes to a model draft, which ranged from 0.175 to 0.448. That paper also flagged the risk that reviewers accept flawed model wording. Both studies use standardized text, where convergence on the model's wording is harmless. In unwritten dialects, convergence on the model's spelling is itself the failure mode, and that is the hypothesis we test.

On the mobile side, Building transcribed speech corpora quickly and cheaply for many languages showed that one-tap phone workflows make collection fast, completing 500 utterances in about 30 minutes, and that read speech deviates from prompts about 10% of the time. That 10% figure is a reminder that a prompt is not a transcript. Lig-Aikuma: A Mobile App to Collect Parallel Speech for Under-Resourced Language Studies supplies the respeaking and free-recording modes we use to gather spontaneous speech.

Research Program

We build a lightweight phone app with two roles. In recording mode, modeled on Lig-Aikuma's free-recording mode, Swiss German speakers record short spontaneous descriptions, which we segment into 2–5 second clips. In transcription mode, a participant hears a clip and types it in their own spelling. Each clip is transcribed by about seven speakers balanced across the seven dialect regions defined in STT4SG-350. We also transcribe a subset of STT4SG-350 test clips, where the Standard German translation provides a semantic anchor.

Each transcriber is randomly assigned, per clip, to one of three conditions. In the unprimed condition there is a blank text box. In the single-suggestion condition the transcriber edits one draft, made by running a fine-tuned XLS-R recognizer through the phoneme-to-grapheme generator. In the multi-candidate condition, each word slot offers three to five tappable spellings in random order, drawn from the generator and from variants attested in SMS4science, plus free editing.

Transcripts of each clip are aligned to a common set of word slots by progressive character alignment. Lexical identity per slot is assigned by clustering spellings with the Standard German anchor where available. Disputed clusters go to a small expert pass.

The analyses follow directly. We compute $\alpha_{\text{lex}}$, $\alpha_{\text{surf}}$, and $D$ per clip and per region pairing, and test whether regional mismatch between speaker and transcriber lowers $\alpha_{\text{lex}}$, as Keleg et al. found for labels. We measure anchoring as the excess rate at which final spellings match a suggested form, relative to that form's base rate among unprimed transcribers. We measure distortion as the divergence between each condition's per-word spelling distribution and the unprimed distribution. We record time per clip and HTER against the shown draft. Finally, we build variant sets only from unprimed transcriptions, so the evaluation reference is never contaminated by the suggestion engine.

Baselines

The naive reference point is single-reference WER against one randomly chosen transcriber's spelling. This is what a corpus with one transcript per clip would give. Its weakness is that the ranking of systems may depend on whose spelling was chosen. We quantify this by resampling the chosen transcriber and reporting how often pairs of recognizers swap ranks.

The strongest existing evaluation is the STT4SG-350 setup: the fine-tuned Wav2Vec 2.0 XLS-R 1B model scored against Standard German. It reports a WER of 14.0% and a BLEU of 74.7 on its test set (STT4SG-350: A Speech Corpus for All Swiss German Dialect Regions). We run it on exactly the STT4SG test clips we re-transcribe. Every recognizer we compare is fine-tuned from the same checkpoint with an identical compute and tuning budget, differing only in target (Standard German versus dialect spelling). Because this metric answers a different question, translation quality rather than dialect transcription, beating it means something specific. Our metric must agree at least as well with native-speaker acceptability judgments of dialect output, while remaining applicable where Standard German scoring cannot be applied at all.

The strongest normalization competitor follows Modeling Dialectal Variation for Swiss German Automatic Speech Recognition. Both hypothesis and reference are mapped to normalized forms, and WER is computed on the normalized forms. That paper reports 13.5% WER on its TV Box corpus, which is not our data, so it serves as context only. We reimplement the normalizer on our lexicon and give it the same seed words.

The most dangerous metric baseline is oracle multi-reference WER: the minimum WER over all k unprimed transcripts. We compare it against our slot-wise variant sets at equal k.

For the collection side, the reference point is the unprimed condition, and the single-suggestion condition is the standard author-reviewer pipeline. The ablation that isolates the key ingredient is multi-candidate versus single-suggestion at matched candidate quality. A second ablation varies the support threshold of the variant set between one and three transcribers, to show that requiring independent attestation is what filters out individual typos.

Success means meeting three conditions. Variant-set WER must correlate with acceptability judgments significantly better than single-reference and normalized WER, and no worse than oracle multi-reference. Multi-candidate must keep spelling-distribution divergence within the range of a split of the unprimed data while recovering at least half of the single-suggestion time saving.

Why not just…?

Why not just bypass dialect spelling and score against Standard German, as STT4SG-350 does? That metric is clean, but it measures translation into a different language. It cannot assess a system that must produce or read the dialect text people actually write, and SMS4science shows that such text is the norm in informal Swiss communication. It also requires prompted or translated speech for the reference, which excludes spontaneous recordings. We keep it as a baseline. If dialect-target systems ranked identically under it and under acceptability judgments, our metric would add little for evaluation, and the experiment would reveal that.

Why not just take the minimum WER over several independent transcripts? This is the real threat, because it already credits valid variants once multiple transcripts exist. Two differences matter. Whole-sentence minimum cannot mix a valid spelling from one transcriber with a valid spelling from another in the same sentence. It also rewards matching a single sloppy transcriber. Slot-wise sets with a two-transcriber support threshold address both. If oracle multi-reference correlates with acceptability as well as our sets do, we will report that honestly. The contribution then narrows to the collection and anchoring findings plus the agreement diagnostics, which remain new because no file paper reports transcriber agreement for dialect speech.

Why not just normalize everything with a phoneme-to-grapheme or normalization model and compute WER on normalized forms? Normalization hides the variation it is meant to evaluate. It also adds its own errors, which are counted as recognizer errors, and nothing in Khosravani et al. validates its mappings against speakers. We test it directly against acceptability judgments. Agreement on lexical identity in our panel also gives an estimate of how often the normalizer merges words that speakers consider different.

Why not just let a large language model generate the spelling variants zero-shot and skip human panels? For the closest analogues in the file, unguided generation fails. Raw generation violated constraints 35.8% of the time before human filtering in Bae et al. Zero-shot translation into unstandardized Middle Eastern varieties scored below 1 BLEU in PARME: Parallel Corpora for Low-Resourced Middle Eastern Languages. We include model-generated variants as one source of candidates in the multi-candidate condition, and report the fraction of them that unprimed speakers ever produce.

Decisive Checks

The study succeeds or fails on four results, all taken from the same pool of clips. The first is whether $D$ is reliably above zero with confidence intervals from clip resampling. If speakers disagree on spelling no more than chance predicts given their agreement on words, the premise collapses. The second is whether variant-set WER reorders a set of six recognizers relative to single-reference WER, and whether that reordering moves toward native speakers' pairwise acceptability judgments, which are collected blind to the metric. The third is the anchoring effect size in the single-suggestion arm and whether the multi-candidate arm reduces it. The fourth is whether regional mismatch between speaker and transcriber predicts lower $\alpha_{\text{lex}}$, which would extend the Arabic dialectness finding from text labels to speech transcription.

There is no reported result on Lacuna for inter-transcriber agreement on Swiss German dialect speech, so every one of these numbers would be a first measurement. That absence is itself the gap this study is designed to fill.