TL;DR
This paper introduces a novel framework for automatically detecting agreement and disagreement in conversational speech by focusing on local acoustic and linguistic features within short speech segments, rather than relying on entire utterances. This approach significantly improves detection accuracy, particularly for conversations where agreement/disagreement is expressed intermittently.
What Problem Does This Paper Address?
Understanding the structure and flow of conversations is crucial for applications like meeting summarization. A key aspect of this is identifying utterances where participants agree or disagree. However, traditional methods often struggle because conversations don't always neatly segment into single-sentence utterances. Participants frequently speak continuously, with agreement or disagreement signals appearing only in parts of a longer utterance. This makes it problematic for methods that assume global features of an entire utterance are representative of its sentiment. The paper tackles this challenge by proposing a way to detect these nuanced signals within smaller, localized segments of speech.
Methods
The proposed framework operates in two main steps:
- Local Class Estimation: This step focuses on short "windows" of speech, typically covering a few words. Within each window, local prosodic (e.g., pitch, intensity, duration) and lexical (e.g., keywords, language model probabilities) features are extracted. These features are then used to estimate the probability (posterior) of the window belonging to one of three classes: agreement, disagreement, or "other." This is a departure from traditional methods that analyze the entire utterance at once.
- Utterance Class Estimation: The posteriors from all the local windows within an utterance are then integrated. This integration considers statistics like the number of windows, class occurrences, and the mean/standard deviation of posteriors. This aggregated information is used to make the final decision about whether the entire utterance is classified as agreement, disagreement, or other.
The novelty lies in the emphasis on local features and the integration strategy, moving away from global utterance-level analysis.
Datasets
The authors collected a new Japanese conversational speech dataset for this research. This dataset consists of over 20 hours of recorded speech from simulated conversations involving 2-4 participants. After segmentation, 2987 utterances were obtained. Crucially, this dataset includes both "utterance labels" (classifying the whole utterance) and "interval labels" (marking specific segments within an utterance that exhibit agreement or disagreement). This interval labeling is key to addressing the problem of partial agreement/disagreement. The dataset's appearance rates for agreement (9%), disagreement (7%), and backchannels (26%) are noted as being similar to those in previous works, suggesting its reliability.
Key Insights and Ideas
- The "Partial Signal" Problem: The most significant insight is the confirmation that agreement and disagreement are often expressed in only parts of an utterance. The dataset analysis showed that for a substantial portion of agreement and disagreement utterances, the relevant signals occupied less than half of the total utterance length. This strongly justifies the need for a local feature-based approach.
- Local Prosody Matters: Prosodic features (pitch, intensity, duration) extracted from short windows proved effective in detecting agreement and disagreement, outperforming a baseline method that used global features.
- Local Lexicon Also Matters: Similarly, lexical features (keywords, language models) also showed significant improvement over the baseline when analyzed locally.
- Integration is Key: While local features are important, the way they are integrated to form an utterance-level decision is critical. The paper demonstrates that combining local estimates leads to better overall performance.
Key Results
- The proposed method significantly improved detection accuracy compared to a baseline using global features.
- Using local prosodic features alone, the total accuracy improved by up to 10.1% over the baseline.
- Using local lexical features alone, the improvement over the baseline was even greater, exceeding 20%.
- Combining both prosodic and lexical features yielded the best results, demonstrating the complementary nature of these cues.
- The optimal local window length was found to be around 3-7 words (self ± 1 to ± 3 words), suggesting that agreement/disagreement signals have a relatively short but not instantaneous duration.
Limitations and Future Work
- Integration Strategy: The authors acknowledge that their current integration method, which uses mean and standard deviation of local posteriors, doesn't fully leverage sequential information. They suggest that more advanced integration techniques could further improve performance.
- Speech Recognition Output: The current experiments used hand-transcribed words. A future direction is to evaluate the system's performance using output from automatic speech recognition (ASR) systems, which is more realistic but introduces ASR errors.
- Dataset Diversity: While the new dataset is valuable, evaluating the method on other conversational speech datasets is suggested.
- Advanced Combinations: The paper notes that the simple combination of prosodic and lexical features might not be optimal, and exploring more sophisticated combination methods, such as variable sub-interval lengths, is a remaining problem.
Summary updated