TL;DR
This paper tackles the challenge of understanding spoken conversations, particularly in noisy call-center environments, by adapting dependency parsing techniques. They demonstrate that by fine-tuning parsers on spontaneous speech data, they can significantly improve the accuracy of identifying semantic roles, paving the way for more robust open-domain spoken language understanding.
What Problem Does This Paper Address?
Extracting meaningful semantic information from spoken conversations, especially those found in customer service call centers, is a significant challenge. These conversations are often characterized by non-canonical language, disfluencies (like repetitions and false starts), transcription errors from Automatic Speech Recognition (ASR) systems, and extraneous information. Traditional parsing methods, designed for clean, written text, struggle with this "noisy" data. The paper aims to bridge this gap by adapting syntactic dependency parsing, a more flexible approach, to better handle the specificities of spontaneous speech for open-domain spoken language understanding (SLU).
Methods
The core of the paper lies in adapting a dependency parser to spontaneous speech. They leverage the MACAON tool suite, which includes a linear-chain CRF POS-tagger and a first-order graph-based dependency parser. The key innovation is the adaptation of this parser. Instead of using a parser trained solely on formal written text (like the French Treebank), they create a specialized training corpus from the RATP-DECODA call-center data. This adaptation involves an iterative process of manual correction of parsing errors on a development set, followed by retraining the parser. This semi-supervised approach allows the parser to learn the specific syntactic structures and disfluencies present in spontaneous speech.
Furthermore, they integrate this adapted parser with a FrameNet-based semantic parsing approach. FrameNet aims to identify "frames" (semantic concepts) and their "roles" (participants in the frame), answering questions like "who does what, where, and when." By merging LU (Lexical Unit) detection with dependency parsing, they can efficiently generate frame candidates from the parsed output, even with noisy ASR transcripts.
Datasets
- French Treebank (FTB): Used as a baseline for training a parser on written text. Specifically, FTB-TRAIN was used for training.
- RATP-DECODA corpus: This is the primary dataset, consisting of 1514 anonymized, transcribed, and annotated conversations from the Paris public transport authority call center. It includes manual annotations for POS tags and syntactic dependencies. The corpus is split into DEC-TRAIN (for training) and DEC-TEST (for evaluation). This corpus is notable for its spontaneous speech characteristics and includes manual annotations of disfluencies.
Key Insights and Ideas
- Spontaneous speech is fundamentally different: The paper strongly emphasizes that parsers trained on formal written text perform poorly on spontaneous speech, even after manually removing disfluencies. This highlights the need for domain-specific adaptation.
- Dependency parsing is more robust for speech: Compared to constituent-based parsing, dependency parsing is better suited for spontaneous speech due to its flexibility in handling ungrammatical structures and its simpler annotation process.
- Semi-supervised adaptation is effective and cost-efficient: The iterative process of manual correction and retraining on a subset of the spontaneous speech data significantly boosts parsing accuracy. This approach is presented as a cost-effective way to obtain specialized training data compared to full manual annotation.
- Merging LU detection with dependency parsing is efficient for FrameNet: This combination allows for the automatic generation of frame candidates, reducing the manual effort required for semantic annotation.
- Even noisy ASR transcripts can yield useful semantic information: Despite high Word Error Rates (WER) in the ASR transcripts, the adapted dependency parser shows a degree of robustness, enabling some level of semantic analysis.
Key Results
- Significant improvement in parsing accuracy: When adapted to the RATP-DECODA corpus, the parser achieved a Labeled Attachment Score (LAS) of 83.86 on spontaneous speech with disfluencies, a substantial improvement over the 58.28 LAS achieved by a parser trained solely on written text.
- Improved semantic dependency detection: The adapted parser demonstrated higher precision (88.2%) in detecting semantic dependency structures compared to the FTB-trained parser (75.9%) on manual transcriptions.
- Robustness on ASR transcripts: While performance was limited on ASR transcripts (e.g., precision of 57.3% for dep1 condition), the adapted parser still showed some capability in detecting semantic dependencies, indicating robustness in challenging conditions.
Limitations and Future Work
- Manual annotation effort: While the adaptation process is more efficient than full manual annotation, it still requires some manual effort for correction and refinement.
- Limited domain specificity: The adapted parser is tailored to the RATP-DECODA corpus. Further work could explore more generalizable adaptation techniques or methods to leverage domain-specific models more effectively for high-WER ASR transcripts.
- Performance on ASR transcripts: The paper acknowledges that performance on ASR transcripts is limited, suggesting that further research is needed to improve robustness in the face of significant transcription errors. Future work could involve integrating parsing more deeply into the ASR process or developing more sophisticated reranking approaches.
Summary updated