Crowdsourcing Swiss Dialect Transcriptions for Assessing Factors in Writing Variations

Simon Clematide, Karina Frick, Noëmi Aepli, Jean-Philippe GoldmanKONVENS 2016 · OpenReview

Abstract

Swiss German lacks a standard orthographic norm, leading to significant variation in written dialect. This paper analyzes existing corpora, identifying limitations in current datasets for understanding the drivers of spelling choices. We propose a crowdsourcing project to collect spontaneous transcriptions from laypeople, aiming to create a benchmark dataset that pairs audio with authentic, non-standardized writing.

Introduction

In German-speaking Switzerland, a "medial diglossia" once defined the linguistic landscape: people spoke dialect but wrote in Standard German. Today, this boundary has collapsed. With the rise of SMS, WhatsApp, and social media, writing in dialect has become the norm for private communication. However, the absence of an official spelling code for Swiss German leads to massive variation; a single word might be spelled in a dozen different ways based on the writer's local dialect, age, or personal preference.

Current research is hampered by "one-sided" datasets. The SMS4science corpus contains authentic, user-generated text but lacks original audio, making it impossible to determine if spelling choices are driven by phonetic nuances. Conversely, the ArchiMob corpus provides audio but uses transcriptions from trained linguists following strict guidelines, which masks how laypeople naturally write.

Methodology

To quantify the "spelling chaos," the researchers employ Shannon entropy ($H$). A word with a strictly normalized spelling has an entropy of 0, while words with numerous variants have high entropy values.

Figure 1: Scatter plot of word frequency (y-axis) vs. entropy (x-axis) in the SMS4science corpus.

Additionally, the Needleman-Wunsch algorithm is used for character-level sequence alignment. This allows researchers to calculate "edit distances" and pinpoint exactly where spellings diverge. To address the data gap, the authors propose a "citizen science" approach, recruiting laypeople to transcribe audio clips "according to their own best practice" via a gaming platform.

Data Analysis and Results

Analysis of the SMS4science corpus reveals that nearly 51% of word types show at least two different spellings. Interestingly, the distribution of entropy in the expert-led ArchiMob corpus and the user-generated SMS4science corpus is remarkably similar, suggesting that underlying dialectal variation "leaks" through even when experts attempt to follow a script.

Table 1: Swiss German spelling variants and their associated entropy (H) in the SMS4science and ArchiMob corpora.

As shown in Table 1, common words like nächste (next) exhibit extreme volatility, with over 25 distinct variants recorded. The data suggests that high usage frequency does not necessarily lead to spelling normalization in a non-standardized language environment.

Discussion and Future Work

Spelling in Swiss German serves as a playground for social identity and phonetic expression. To overcome the "transcription bottleneck," the authors suggest breaking audio into short "prosodic phrases" (2-5 seconds) to reduce cognitive load for volunteers.

Future work involves launching the web-based transcription platform and correlating spelling variations with sociolinguistic metadata, such as the age and home dialect of the transcriber. Ultimately, this multi-transcription dataset will be used to train NLP models for the automatic normalization of dialect text into Standard German.

Summary updated

BibTeX
@inproceedings{clematide2016crowdsourcing,
  author = {Simon Clematide and Karina Frick and Noëmi Aepli and Jean-Philippe Goldman},
  title = {{Crowdsourcing Swiss Dialect Transcriptions for Assessing Factors in Writing Variations}},
  booktitle = {KONVENS},
  year = {2016},
  url = {https://openreview.net/forum?id=a90a433vARY},
}
Figure