TL;DR
Spoken language dependency parsing significantly lags behind written language parsing. This paper investigates how data representations, specifically transcription and annotation choices, impact this gap, finding that utterance segmentation is the primary culprit. By carefully modifying spoken data representations, the authors significantly improve parsing performance, bringing it closer to written language levels.
What Problem Does This Paper Address?
Despite advancements in natural language processing, systems that parse spoken language dependency structures consistently underperform compared to those handling written text. This discrepancy is particularly evident when using the Universal Dependencies (UD) annotation scheme, as seen in the Spoken Slovenian Treebank (SST) versus its written counterpart. The authors aim to pinpoint the specific characteristics of spoken language data that cause this performance drop and explore how modifying these representations can bridge the gap. This is crucial for developing more robust and accurate NLP systems that can effectively process the ever-growing volume of spoken data.
Methods
The researchers employed two state-of-the-art dependency parsers, UDPipe 1.2 (transition-based) and Stanford parser (graph-based), to evaluate the impact of various data modifications. Their approach involved systematically altering the Spoken Slovenian Treebank (SST) in several ways:
- Training Data Modifications: They experimented with training parsers on spoken data alone, written data alone, and a combination of both. They also created versions of the written data that excluded punctuation and used only lowercase to match the spoken data's format.
- Transcription Modifications: This involved altering how spoken utterances were represented. Key changes included:
- Segmentation: Splitting utterances into smaller units (e.g., clauses) or merging them into larger ones (e.g., entire turns).
- Disfluency Removal: Removing filled pauses, repetitions, and other speech errors.
- Discourse Marker Removal: Excluding interjections and politeness expressions.
- Prosody Removal: Excluding markers for intonation, pauses, and vocal sounds.
- Spelling: Using pronunciation-based spelling instead of standard orthography.
- Annotation Modifications: They explored changes to the UD annotation scheme itself, such as:
- Extensions: Replacing specific UD relation subtypes with their universal counterparts.
- Head Attachment: Altering how certain elements (like prosody markers or discourse fillers) attach to the dependency tree.
- Reparandum Attachment: Modifying how edited speech segments are linked.
- Discourse Marker Labeling: Changing the label for clausal discourse markers.
Datasets
The primary dataset used is the Spoken Slovenian Treebank (SST). This is the first spoken treebank annotated with the Universal Dependencies (UD) scheme. For comparison, the authors also used the written Slovenian treebank (SSJ). The SST treebank was derived from the Gos reference corpus of Spoken Slovenian.
Key Insights and Ideas
- Segmentation is King (or Queen): The most significant factor hindering spoken language parsing performance is utterance segmentation. Even when parsing raw transcriptions, the inability to correctly segment utterances into meaningful units causes a substantial drop in accuracy.
- Disfluencies are Less of a Bottleneck Than Expected: While removing disfluencies generally improves performance, it's not the primary driver of the gap. The impact is less pronounced than that of segmentation.
- Prosody Markers Can Help: Surprisingly, removing prosody markers (like intonation or pauses) damages parsing results for both systems. This suggests these markers, often overlooked, can provide valuable cues for parsing.
- Graph-Based Parsers Benefit from Mixed Data: For more complex graph-based parsers (like Stanford), combining spoken and written training data significantly boosts performance, suggesting they can better leverage richer contextual information.
- UD Adaptations are a Good Starting Point: The original UD adaptations for spoken language phenomena in SST provide a reasonable foundation, and further modifications offer incremental improvements rather than revolutionary leaps.
Key Results
- Baseline Gap: The initial gap in LAS (Labeled Attachment Score) between parsing the spoken SST and written SSJ treebanks was substantial, around 30 percentage points.
- Segmentation Impact: Parsing raw text with UDPipe and Stanford parsers showed significantly worse performance on SST compared to SSJ. When only dependency parsing was considered (with gold-standard segmentation), the gap narrowed considerably, highlighting segmentation's critical role.
- Best Combination Performance: By combining several beneficial modifications (e.g., improved segmentation, removal of disfluencies and discourse markers, and specific annotation changes), the authors achieved LAS scores of 79.58% for UDPipe and 87.35% for Stanford.
- Closing the Gap: These best-performing models significantly narrowed the LAS gap to 4.63% for UDPipe and 2.25% for Stanford compared to their same-size written counterparts.
Limitations and Future Work
- System Dependency: The effectiveness of certain data modifications varied between the UDPipe and Stanford parsers, indicating that parser architecture plays a role.
- UD Adaptations: While the original UD adaptations for spoken language are a good starting point, the paper explored only a few specific transformations. Further evaluation of UD principles for spoken language is needed.
- Future Work: The authors suggest increasing the size of annotated spoken data, reconsidering transcription and annotation principles for better segmentation and disfluency handling, and evaluating these findings against recent speech-specific parsing systems. They also propose exploring cross-domain strategies for parsing noisy data and conducting more granular analyses of how specific dependency relations are affected by these modifications.
Summary updated