Abstract
Supervised dialogue management models often struggle with small, imbalanced datasets and out-of-vocabulary (OOV) words. This study investigates a combination of automated data expansion through back-translation paraphrasing and runtime lexical substitution using semantic resources. By integrating these methods with lemmatization, the researchers achieved an 8.9% improvement in dialogue act classification accuracy. The results underscore the necessity of linguistic preprocessing and data balancing for robust situated dialogue systems.
Introduction
Supervised models for dialogue management—the components that determine how a virtual character should respond—face a significant bottleneck: the need for large, balanced, and varied training datasets. Human-annotated data is typically scarce and highly skewed, leading to models that fail when encountering OOV words or unfamiliar phrasing. This paper addresses these challenges by exploring automated methods to increase model robustness without extensive manual annotation.
Methods
The researchers implemented a three-pronged strategy to improve a logistic regression classifier used for dialogue act labeling:
- Automated Paraphrasing (Data Augmentation): To expand the training set, they used a "pivot machine translation" approach. They took original French utterances and translated them into six different languages and then back into French using the Google Translate API. This captured both lexical and syntactic variety automatically.
- Smart Data Balancing: Simply adding paraphrases isn't enough if the data remains skewed. They developed an algorithm that adds more paraphrases to underrepresented categories and fewer to overrepresented ones, creating a more balanced training set.
- Lexical Substitution at Runtime: To handle unknown words during actual use, the system looks for synonyms in external resources like EuroWordNet (EWN), a custom synonym lexicon (DIC), or via Distributional Semantics (RI). If an unknown word is found, it is replaced with a known synonym from the training data before classification.
- Lemmatization: They compared training on raw word forms versus "lemmas" (the dictionary base form of a word), using the TreeTagger tool. This proved to be a critical enabler for synonym matching.
Datasets
The experiments were conducted using data from a "serious game" called Mission Plastechnologie.
- H-H Corpus: Approximately 1,250 Human-Human dialogues manually annotated with 28 different dialogue acts. This served as the primary training and cross-validation set.
- H-C Corpus: A separate test set of 550 Human-Computer dialogues. This represents a more difficult "real-world" evaluation where users often speak differently than they do to other humans.
Results and Discussion
The combination of all three techniques (paraphrasing, synonym handling, and lemmatization) yielded the highest gains: +8.9 percentage points in accuracy for the H-H cross-validation and +2.3 points for the H-C evaluation. On its own, data expansion via paraphrasing provided a significant boost of up to 8.27 points when combined with lemmatization.
Key insights include:
- Lemmatization is the Enabler: Synonym substitution is almost useless without lemmatization. Because synonym dictionaries and WordNet typically store base forms, the system needs to reduce both the user's input and the training data to lemmas to find a match.
- The "Weak Out-of-Domain" Effect: Performance consistently drops when moving from Human-Human (H-H) data to Human-Computer (H-C) data. This suggests that people adapt their language when they know they are talking to a machine, creating a distribution shift.
- Synonym Resolution: The system successfully resolved about 50% of unknown words encountered at runtime. Of the remaining 50%, 17% were "ill-formed" (typos/slang) and 33% simply had no match in the lexical resources.
Conclusion
This research demonstrates that automated data augmentation and semantic substitution can significantly improve dialogue act classification. While a performance gap remains between human-human and human-computer interactions, the proposed techniques provide a scalable path toward more robust and flexible dialogue systems. Future work will focus on better handling of "ill-formed" words and incorporating conversational context to further refine performance.
Summary updated