TL;DR
This paper explores using Latent Semantic Analysis (LSA), a technique that models word meaning based on co-occurrence, for classifying dialogue acts in conversations. They found that plain LSA performs well, and while augmenting it with additional linguistic features didn't consistently improve results, it showed promise for smaller datasets.
What Problem Does This Paper Address?
Understanding the role of an utterance within a dialogue is crucial for building intelligent dialogue systems. This "dialogue act classification" helps systems interpret user intent (e.g., asking a question, making a request) and generate appropriate responses. While various empirical methods exist, this paper investigates Latent Semantic Analysis (LSA) as a novel approach. LSA, which represents words and passages in a semantic space based on their co-occurrence, has been successful in other NLP tasks but hasn't been widely applied to dialogue act classification. A key question is whether LSA, relying solely on raw text, can be improved by incorporating other linguistic information.
Methods
The authors experiment with four LSA-based methods:
- Plain LSA: This is the standard LSA approach. It builds a "Word-Document" matrix where rows represent words and columns represent documents (dialogue turns). LSA then uses Singular Value Decomposition (SVD) to reduce the dimensionality, creating a semantic space where word and document vectors represent their meaning. Classification is done by comparing the test document's vector to training document vectors and assigning the most frequent tag of the closest match.
- Clustered LSA: Instead of a Word-Document matrix, this method uses a "Word-Tag" matrix, where columns represent dialogue act types. This allows for faster classification as test documents are only compared to tag vectors.
- Semi-Clustered LSA: This method aims to retain document distribution information lost in Clustered LSA. It uses the SVD from the Word-Document matrix to reduce the Word-Tag matrix to the same k-dimensional space.
- Feature LSA (FLSA): This approach augments plain LSA by incorporating additional features beyond just words. Specifically, they experiment with including the previous n dialogue acts as features, creating a combined "Word-Tag-Document" matrix.
Datasets
The paper uses two corpora:
- DIAG: A newly introduced corpus of computer-mediated tutoring dialogues. It consists of 23 dialogues, 607 unique words, and 660 dialogue acts, annotated with four main dialogue acts: problem solving, judgement, domain knowledge, and other.
- Spanish CallHome: A corpus of 128 unrestricted Spanish phone calls, totaling 12066 words and 44628 dialogue acts. This corpus has a richer annotation scheme, leading to many dialogue act tags. For experiments, they used collapsed versions: CallHome37 (37 tags) and CallHome10 (8 dialogue acts plus two noise tags).
Key Insights and Ideas
- LSA's inherent potential: Plain LSA, despite its "bag-of-words" nature, demonstrates a strong baseline performance for dialogue act classification, suggesting it captures significant semantic information relevant to this task.
- Dataset size matters for LSA variants: Clustered and semi-clustered LSA performed poorly on the larger CallHome corpus but showed improvement on the smaller DIAG corpus. This suggests these methods might be more effective when the number of documents or target classifications is limited.
- Augmentation challenges: The hypothesis that adding features like previous dialogue acts to LSA (FLSA) would consistently improve performance was not strongly supported by their experiments. Adding more than one previous dialogue act did not yield better results.
- Comparison to prior work: The plain LSA results on CallHome are comparable to published results using more complex methods like HMMs and neural networks, even though those methods often incorporate richer features (e.g., part-of-speech tags).
Key Results
- Plain LSA Performance: On the DIAG corpus, plain LSA achieved 75.73% accuracy. On CallHome37, it achieved 65.36%, and on CallHome10, 68.91%.
- Clustered/Semi-Clustered LSA: Semi-Clustered LSA improved plain LSA by 3% on DIAG. However, both clustered methods performed significantly worse than plain LSA on CallHome37 and CallHome10.
- FLSA Performance: FLSA did not consistently outperform plain LSA. For DIAG, FLSA achieved 74.26% accuracy, slightly lower than plain LSA. For CallHome, FLSA achieved 62.59% (CallHome37) and 66.57% (CallHome10), also generally lower than plain LSA.
- Baseline Comparison: In all cases, LSA methods outperformed a simple baseline of assigning the most frequent dialogue act.
Limitations and Future Work
- FLSA limitations: The authors acknowledge that their FLSA experiments were limited in the features they incorporated (only previous dialogue acts). They suggest that other features, such as syntactic roles, might be more beneficial.
- Corpus dependency: The performance of clustered and semi-clustered LSA appears to be highly dependent on the size of the corpus and the target classification. Further investigation into this correlation is needed.
- Future Directions:
- Investigate the relationship between corpus size/classification complexity and the performance of clustered LSA.
- Incorporate other features into FLSA, such as syntactic roles.
- Test LSA and FLSA on different corpora, like the Map Task corpus, which has richer syntactic annotations.
- Explore FLSA's applicability to other tasks, such as assessing text coherence.
- Future work could also explore incorporating part-of-speech tags, which were used in prior work but not in this study.
Summary updated