TL;DR
This paper tackles the challenge of making semantic parsers, which understand the meaning of text, work well with spoken language, especially when dealing with errors from automatic speech recognition (ASR). They adapt a technique called adversarial learning, originally used for text-based domain adaptation, to improve how well a FrameNet semantic parser generalizes to spoken data, including noisy ASR transcripts.
What Problem Does This Paper Address?
Extracting structured meaning from text, known as semantic parsing, is crucial for Spoken Language Understanding (SLU). A common pipeline involves an Automatic Speech Recognition (ASR) system converting speech to text, followed by a semantic parser. However, semantic parsers trained on clean, written text often perform poorly on spoken language, particularly when that speech has been processed by an ASR system, introducing errors. These errors, along with stylistic differences between written and spoken language, create a significant "domain gap." While methods exist to simulate ASR errors in training data, they don't fully address the inherent differences in spoken language. This paper aims to bridge this gap by adapting adversarial learning, a technique that encourages models to learn domain-invariant representations, to improve semantic parsing robustness for spoken language understanding.
Methods
The core of the paper is the adaptation of a FrameNet semantic parser using adversarial learning.
- Base Model: They use a bi-directional Gated Recurrent Unit (biGRU) based sequence tagger for semantic frame parsing. This model performs frame selection and argument classification in a single step and is trained with rich features, including pre-trained word embeddings. They also employ a BIO label encoding and an A* decoding strategy for coherence. A hyper-parameter
δis used to tune the precision/recall trade-off. - Adversarial Domain Classifier: The novel aspect is the integration of an adversarial domain classifier. This classifier is added on top of the biGRU's last hidden layer, connected via a gradient reversal layer. The goal is to train the semantic parser to be "domain-independent." The semantic parser is trained to minimize its primary task loss (frame and argument identification), while the adversarial classifier is trained to distinguish between different data sources (e.g., written text vs. spoken transcripts). The adversarial loss is subtracted from the main task loss, effectively training the parser to "fool" the domain classifier.
- Training Stability: To ensure stable training, the adversarial gradients are attenuated by a factor
λ, which increases gradually over epochs.
Datasets
- CALOR Corpus: A French corpus of encyclopedic written documents (from Wikipedia, Vikidia, and ClioTexte) annotated with FrameNet. This serves as the "written" domain.
- PODCAST Corpus: A compilation of French radio podcast transcriptions from "Les P'tits Bateaux." This corpus represents the "spoken" domain and was manually transcribed and annotated with FrameNet.
- PODCAST-ASR Corpus: The PODCAST corpus processed by an ASR system (Cobalt Speech Recognition) to simulate real-world spoken language understanding scenarios with transcription errors.
The paper notes that for their specific Information Extraction tasks, they used a partial parsing policy, annotating only 53 Frames, which allows for more occurrences per frame compared to the full FrameNet.
Key Insights and Ideas
- Adversarial Learning for Spoken Language: The paper demonstrates that adversarial learning, previously successful for domain adaptation in written text, can be effectively transposed to spoken language adaptation for semantic parsing.
- Robustness to ASR Errors: The adversarial approach significantly improves performance on ASR-generated transcripts, suggesting it helps build representations that are less sensitive to transcription noise.
- Generalization Beyond Lexical Bias: The method helps overcome biases learned from specific data sources, leading to models that are more robust to variations in elocutionary style and lexical usage.
- Spoken Language Complexity: Processing "simplified" language for children (as in the PODCAST corpus) can be surprisingly challenging for semantic parsers due to idioms, oral expressions, and polysemous words that don't map directly to semantic frames.
- Argument Identification is Harder: ASR errors have a more detrimental impact on argument identification than frame identification, as they often affect short, syntactically crucial words (pronouns, prepositions, articles) and can introduce syntax errors.
Key Results
- Improved Performance on ASR Data: The adversarial approach (biGRU+adv) consistently outperforms the baseline biGRU model on the PODCAST-ASR test set for both Frame Identification (FI) and Argument Identification (AI). The improvement for FI on ASR data was up to +2.5 points.
- Robustness to Domain Shift: The adversarial model showed improved generalization capabilities on both manual (PODCAST-GOLD) and automatic (PODCAST-ASR) speech transcripts compared to models trained solely on written text (CALOR).
- Handling Difficult Cases: Adversarial training showed larger gains on more complex scenarios, such as identifying non-core Frame Elements, nominal triggers, and in longer sentences.
- WER Impact: Performance degrades with increasing Word Error Rate (WER) in ASR transcripts, but the adversarial learning strategy is more beneficial for transcriptions with higher WER.
Limitations and Future Work
- Short Sentences: Adversarial learning slightly harmed performance on short sentences in their analysis.
- Partial Parsing: The use of a partial parsing policy (53 Frames) might limit the generalizability to the full FrameNet annotation scheme.
- Future Work: The authors suggest that their approach could be applied to more Spoken Language Understanding tasks, implying further exploration in areas like intent recognition or dialogue state tracking. They also highlight the need for better metrics to tune ASR systems for downstream SLU tasks.
Summary updated