TL;DR
This paper proposes using Word Sense Disambiguation (WSD) as a crucial component for evaluating human-computer spontaneous conversation, moving beyond surface-level response quality to assess genuine mutual understanding. The authors empirically validate this approach by analyzing WordNet's coverage of conversational language and evaluating state-of-the-art WSD models on naturalistic dialog data.
What Problem Does This Paper Address?
Current methods for evaluating social dialog systems, like chatbots, primarily focus on the quality of the system's generated text. This often means checking if responses are relevant or grammatically correct, but it doesn't truly capture whether the human and the computer understand each other. This "Great Misalignment Problem" hinders the development of truly collaborative AI systems. For instance, a chatbot might give an inappropriate response, but it's unclear if this is due to a misunderstanding of a specific word's meaning or a general lack of conversational ability. The authors argue that to improve usability, evaluations need to incorporate human-verifiable aspects of language competence, specifically focusing on how systems interpret ambiguous words.
Methods
The core of this research is the proposal and empirical validation of Word Sense Disambiguation (WSD) as a key evaluation metric. The authors investigate three main research questions:
- WordNet Coverage: How well does WordNet, a popular English sense inventory, cover the meanings of words used in spontaneous dialog?
- SOTA WSD Model Readiness: Are current state-of-the-art WSD models, trained on general data, ready for conversational text?
- Contextual Embeddings Effectiveness: How effective are contextual embeddings (like BERT) for WSD in spontaneous conversation?
To address these, they:
- Automatically annotated ambiguous words in a naturalistic dialog corpus (NEWT-SBCSAE) using a SOTA WSD model (AMuSE-WSD) and a baseline model based on contextual embeddings.
- Collected human judgments on the WSD model outputs and the appropriate senses.
- Evaluated WordNet's coverage and the efficacy of WSD models using this human-annotated data.
Datasets
- NEWT-SBCSAE: A publicly accessible corpus of naturalistic spontaneous dialogs in American English, used for both automatic annotation and human evaluation. It's a subset of the Santa Barbara Corpus of Spoken American English.
- WordNet 3.0: Used as the sense inventory for WSD.
- SemCor corpus: Used for training the AMuSE-WSD model.
Key Insights and Ideas
- WSD as a Bridge: WSD offers a concrete, human-verifiable way to assess mutual understanding, moving beyond subjective ratings of overall response quality.
- Transparency is Key: By exposing the chatbot's internal interpretation of ambiguous words, evaluators can provide more targeted and useful feedback, improving the development cycle.
- WordNet's Robustness: WordNet, despite its age, demonstrates good coverage (96.3%) of ambiguous words found in spontaneous conversations, suggesting it's a viable resource for this task.
- Contextual Embeddings Shine: The study highlights the potential of contextual embeddings from models like BERT for WSD, showing they can be effective even in the less structured domain of spontaneous dialog.
Key Results
- WordNet Coverage: WordNet covers 96.3% of ambiguous words in the NEWT-SBCSAE corpus, with good coverage across different parts of speech (adjectives, adverbs, nouns, verbs).
- SOTA WSD Model Performance (AMuSE-WSD): The AMuSE-WSD model achieved 73.7% accuracy on conversational text, performing consistently across different parts of speech. This is lower than its performance on non-conversational data but still encouraging.
- Baseline WSD Model Performance (DistilBERT): The DistilBERT-based baseline model achieved a 36% accuracy, which is considered encouraging given the average of 9.9 WordNet senses per word token. Its performance was lower on verbs, likely due to their higher sense average.
- Transformer Variants: Comparing BERT, XLNet, and RoBERTa for the baseline model, BERT generally performed best, followed by RoBERTa, with XLNet showing lower performance.
- Annotation Challenges: The study noted that aligning off-the-shelf WSD model outputs with NLP pipelines can be challenging, with 5.5% of tokens not being directly alignable.
Limitations and Future Work
- Sense Inventory Limitations: Relying solely on WordNet means the evaluation might miss semantically ambiguous words not present in WordNet or inherit WordNet's granularity issues. Natural language is constantly evolving, and sense inventories struggle to keep pace.
- End-to-End Models: The proposed WSD evaluation method may be difficult to adopt for "black box" end-to-end dialog systems, as their internal workings are not easily interpretable at a word level.
- Future Work: The authors plan to conduct a detailed analysis of annotator disagreements and model errors, explore the impact of input utterance manipulation on WSD performance, and investigate the distinction between polysemous and homonymous senses (which WordNet doesn't explicitly handle). They also envision coupling WSD with a reasoning judgment component for a more comprehensive evaluation of mutual understanding.
Summary updated