Abstract
Swiss German lacks a standard orthographic norm, leading to significant variation in written dialect. This paper analyzes existing corpora, identifying limitations in current datasets for understanding the drivers of spelling choices. We propose a crowdsourcing project to collect spontaneous transcriptions from laypeople, aiming to create a benchmark dataset that pairs audio with authentic, non-standardized writing.
Introduction
In German-speaking Switzerland, a "medial diglossia" once defined the linguistic landscape: people spoke dialect but wrote in Standard German. Today, this boundary has collapsed. With the rise of SMS, WhatsApp, and social media, writing in dialect has become the norm for private communication. However, the absence of an official spelling code for Swiss German leads to massive variation; a single word might be spelled in a dozen different ways based on the writer's local dialect, age, or personal preference.
Current research is hampered by "one-sided" datasets. The SMS4science corpus contains authentic, user-generated text but lacks original audio, making it impossible to determine if spelling choices are driven by phonetic nuances. Conversely, the ArchiMob corpus provides audio but uses transcriptions from trained linguists following strict guidelines, which masks how laypeople naturally write.
Methodology
To quantify the "spelling chaos," the researchers employ Shannon entropy ($H$). A word with a strictly normalized spelling has an entropy of 0, while words with numerous variants have high entropy values.
Additionally, the Needleman-Wunsch algorithm is used for character-level sequence alignment. This allows researchers to calculate "edit distances" and pinpoint exactly where spellings diverge. To address the data gap, the authors propose a "citizen science" approach, recruiting laypeople to transcribe audio clips "according to their own best practice" via a gaming platform.
Data Analysis and Results
Analysis of the SMS4science corpus reveals that nearly 51% of word types show at least two different spellings. Interestingly, the distribution of entropy in the expert-led ArchiMob corpus and the user-generated SMS4science corpus is remarkably similar, suggesting that underlying dialectal variation "leaks" through even when experts attempt to follow a script.
As shown in Table 1, common words like nächste (next) exhibit extreme volatility, with over 25 distinct variants recorded. The data suggests that high usage frequency does not necessarily lead to spelling normalization in a non-standardized language environment.
Discussion and Future Work
Spelling in Swiss German serves as a playground for social identity and phonetic expression. To overcome the "transcription bottleneck," the authors suggest breaking audio into short "prosodic phrases" (2-5 seconds) to reduce cognitive load for volunteers.
Future work involves launching the web-based transcription platform and correlating spelling variations with sociolinguistic metadata, such as the age and home dialect of the transcriber. Ultimately, this multi-transcription dataset will be used to train NLP models for the automatic normalization of dialect text into Standard German.
Summary updated