TL;DR
This paper investigates whether removing conversational "filler" words (like "um," "uh," "you know") before parsing speech transcripts improves parsing accuracy. They found that while this "early deletion" offers only modest gains for in-domain parsing, it significantly boosts performance when adapting parsers trained on formal text (like news articles) to understand spoken language.
What Problem Does This Paper Address?
Processing spoken language with tools designed for written text presents a significant challenge. Conversational speech is rife with disfluencies – speech repairs and filler words – that can introduce ambiguity and confuse standard parsing algorithms. While previous work showed that removing "speech repairs" (e.g., "I went to the store... I mean, I went to the mall") before parsing could improve accuracy, the impact of removing "fillers" (e.g., "you know," "like") remained less clear. This paper aims to fill that gap by systematically evaluating the benefit of early filler deletion on parsing accuracy, particularly in the context of adapting text-based parsers to spoken language.
Methods
The core idea is to preprocess conversational speech transcripts by removing filler words before feeding them into a parser. The authors use a state-of-the-art parser (Charniak, 2000) and employ an automatic filler detection system (Johnson et al., 2004) to identify and remove these disfluencies. They then reinsert the deleted fillers as a special "FILLER" constituent in the parsed output.
The evaluation focuses on two main scenarios:
- In-domain parsing: Training and testing the parser on conversational speech data.
- Out-of-domain parsing (adaptation): Training the parser on formal text (Wall Street Journal corpus) and testing it on conversational speech. This scenario is crucial for understanding how well text-based tools can be adapted to spoken language.
They also utilize two evaluation metrics:
- PARSEVAL: A standard metric for evaluating the accuracy of syntactic parse trees.
- Dependency-based evaluation: A metric that focuses on the relationships between words, which can offer complementary insights.
Datasets
- Switchboard Corpus: Used for training and evaluating parsing of conversational speech. This corpus is annotated with both syntactic structure and disfluency information.
- Wall Street Journal (WSJ) Corpus: Used for training the parser in the out-of-domain (text-only) condition.
Key Insights and Ideas
- Fillers are a nuisance for parsers: Even when listeners can easily disambiguate meaning, filler words can create syntactic ambiguity that hinders automated parsing.
- Early deletion is a form of "bleaching": Removing fillers is a way to make spoken language transcripts more closely resemble clean text, which is beneficial for text-based processing tools.
- Adaptation is where early deletion shines: The most significant gains in parsing accuracy come when adapting a parser trained on formal text to spoken language. This suggests that filler removal is a key step in bridging the gap between text and speech processing.
- Fillers are distinct from repairs: While both are disfluencies, the paper implies that their impact and the effectiveness of their removal might differ, with fillers being a more persistent challenge for text-based parsers adapting to speech.
Key Results
- In-domain parsing: Early deletion of fillers yielded only a modest improvement in parsing accuracy (e.g., from 87.8% to 88.9% on the bracket-based metric).
- Out-of-domain parsing (adaptation): This is where the benefits were substantial. Early deletion of fillers improved parsing accuracy by 5-10% when adapting a text-trained parser to speech.
- Filler detection accuracy: The system was able to identify filler words with a high F-score (93.1%), indicating that the detection mechanism itself is quite effective.
Limitations and Future Work
- In-domain gains are limited: The authors acknowledge that early deletion of fillers has limited potential to improve parsing accuracy within conversational speech data compared to what has been seen with speech repairs.
- Text-trained parsers still lag: Even with oracle knowledge of disfluencies (meaning perfect removal), the text-trained model still significantly underperformed the in-domain model. This indicates that additional methods for "bleaching" speech data are still needed.
- Future directions: The authors suggest exploring alternative methods for integrating detected filler information (e.g., parse reranking) and evaluating the interaction with Automatic Speech Recognition (ASR) errors and sentence boundary detection. They also plan to investigate the benefit of disfluency modeling for text-based machine translation of speech data.
Summary updated