TL;DR
This paper tackles the challenge of training Spoken Language Understanding (SLU) systems with limited labeled data, especially in the presence of noisy speech. They propose two novel active learning methods, Weighted Gradient Uncertainty (WGU) and Strict Local Density (SLD), which outperform existing techniques in both noisy and clean conditions, with SLD showing a slight edge.
What Problem Does This Paper Address?
Spoken Language Understanding (SLU) systems aim to extract meaning from spoken words, powering applications like voice assistants. A major hurdle in developing these systems is the high cost and time involved in manually labeling spoken utterances. Active learning offers a solution by intelligently selecting the most informative unlabeled data points for human annotation, thereby reducing the overall labeling effort. However, SLU systems are particularly susceptible to noise (e.g., from Automatic Speech Recognition errors or user mistakes), and existing active learning methods haven't been thoroughly evaluated or adapted for these noisy conditions. This paper addresses this gap by comparing various active learning strategies in noisy SLU scenarios and introducing new methods designed to handle this challenge.
Methods
The paper evaluates several established active learning strategies, including:
- Uncertainty Sampling: Queries instances where the model is least confident about the label (Least Confident and Sequence Entropy methods).
- Query-By-Committee (QBC): Uses a committee of models trained on different subsets of data to identify instances where the committee members disagree the most.
- Fisher Information Ratio (FIR): Measures informativeness based on the Fisher information matrix, aiming to select instances that maximally reduce model uncertainty.
- Instability Sampling (LIIS/LSIS): Selects instances that are "unstable" across learning cycles, meaning their predicted labels change significantly.
The authors then propose two novel methods:
- Weighted Gradient Uncertainty (WGU): An improvement on the FIR method, WGU leverages the Fisher information framework but proposes a new measure that favors instances with high uncertainty across different dimensions of the CRF feature space.
- Strict Local Density (SLD): This method addresses noisy data by introducing the concept of "local density." Instead of relying on global density, it considers the density of an instance relative to its k-nearest neighbors. SLD first filters out instances with low local density (potentially noisy or outliers) and then applies an uncertainty-based query strategy to the remaining instances.
Datasets
The experiments were conducted on the ATIS (Air Travel Information System) corpus, specifically the class-A utterances from the ATIS-3 corpus. This dataset contains utterances related to flight, airport, and airline information, along with corresponding SQL queries that were semi-automatically converted into flat semantic representations. The authors also describe a noise model based on actual human and ASR errors observed in ATIS data to simulate noisy conditions.
Key Insights and Ideas
- Noise is a critical challenge for SLU active learning: Existing methods, while effective in clean data, struggle when faced with noisy speech, highlighting the need for noise-aware strategies.
- Local density is key to handling noise: The concept of local density, focusing on an instance's neighborhood rather than the entire dataset, proves effective in identifying and avoiding noisy or outlier instances.
- Strict filtering of low-density instances is beneficial: The SLD method's approach of first removing low-density instances before applying uncertainty sampling is more effective than simply multiplying density with uncertainty.
- WGU offers a theoretically grounded alternative: While FIR has theoretical appeal, WGU provides a practical implementation that performs well, particularly in the initial stages of active learning.
Key Results
- Both proposed methods, WGU and SLD, outperformed existing active learning methods in both noise-free and noisy conditions.
- SLD was found to be slightly superior to WGU, achieving the best overall performance, especially in noisy scenarios.
- In noise-free conditions, SLD improved upon the base uncertainty measure (LC) for almost all degrees of locality.
- In noisy conditions, SLD showed a more significant improvement over LC, demonstrating its effectiveness in filtering out noisy instances.
- The instability sampling methods (LIIS, LSIS) showed some improvement over uncertainty sampling but were not as effective as LD or SLD.
- Query-by-committee (QBC) was the slowest method, while LC was the fastest. WGU was also noted to be relatively slow.
Limitations and Future Work
- Computational Cost: Methods that iterate over labelings (like QBC and WGU) tend to be slower than simpler methods like LC. The paper notes that WGU is "rather slow in comparison to LC."
- Future Work: The authors suggest that applying local density to other methods, such as WGU, could lead to even better results. They also mention that the strict application of local density (SLD) is more effective than traditional multiplicative approaches, implying further exploration of density-based filtering could be fruitful.
Summary updated