Linguistically-Informed Specificity and Semantic Plausibility for Dialogue Generation

Wei-Jen Ko, Greg Durrett, Junyi Jessy LiNAACL 2019 · OpenReview

Abstract: Wei-Jen Ko, Greg Durrett, Junyi Jessy Li. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).

2019.

Introduction

Open-domain dialogue systems based on standard sequence-to-sequence (Seq2Seq) architectures frequently suffer from the "boring chatbot" problem. These models tend to favor generic, high-probability responses such as "I don't know" or "I'm not sure," which are prevalent in training datasets. While previous attempts to mitigate this have focused on penalizing common words through frequency-based metrics, such approaches often trigger a specificity-plausibility trade-off. The authors discovered that approximately 33% of specific responses from standard models are semantically implausible. Forcing a model to be highly specific can lead to "hallucinated" or logically inconsistent outputs—for instance, a model might claim to be a millionaire while simultaneously stating it cannot afford basic expenses. This research proposes a method to enhance informativeness without compromising the logical and grammatical integrity of the conversation.

Methods

The proposed solution consists of two primary components: explicit specificity conditioning and a semantic plausibility reranker.

1. Conditioning on Specificity

Rather than relying solely on next-word prediction, the model's decoder is conditioned on a target "level" of specificity. The researchers define these levels using three distinct metrics:

  • Normalized Inverse Word Frequency (NIWF): A traditional metric based on word rarity.
  • Perplexity Per Word (PPW): An information-theoretic measure of sentence content.
  • Linguistically-Informed Specificity: A novel metric that evaluates sentence structure, entity presence, and "instantiation" (the degree of detail provided for a previous concept).

During training, the model learns to associate these metrics with response generation. At inference, the specificity level can be set to its maximum to encourage detailed responses.

The model architecture integrates specificity levels into the decoding process.

2. Semantic Plausibility Reranking

To address the nonsensical outputs often generated by high-specificity models, the authors developed four classifiers to identify common semantic and grammatical errors. These classifiers were trained using a synthetic data augmentation strategy, where "correct" sentences were systematically "broken":

  • Phrase Compatibility: Swapping phrases between sentences to detect contradictions.
  • Content Word Plausibility: Replacing nouns or verbs with random words of the same part-of-speech.
  • Discourse Connectives: Swapping logical connectors like "but" and "and."
  • Cohesion and Grammar: Swapping function words such as pronouns and prepositions.

The final system generates multiple candidate responses and uses these classifiers to select the most plausible one.

Results and Evaluation

The model was evaluated on two benchmarks: OpenSubtitles (diverse movie dialogue) and PersonaChat (consistent persona-based dialogue).

Human evaluation and automatic metrics demonstrate the effectiveness of the proposed model.

Key findings include:

  • Improved Informativeness: On PersonaChat, the full model (+plausibility) significantly outperformed the Seq2Seq baseline in informativeness (3.39 vs 3.01) and topic relevance (3.45 vs 2.84).
  • Plausibility Recovery: While high specificity initially reduced human-rated plausibility to 65.2% on PersonaChat, the reranking system successfully recovered this score to 71.7%.
  • Linguistic Superiority: Models utilizing linguistically-informed metrics were judged more informative by humans than those relying on simple word frequency (NIWF).

Discussion and Conclusion

This work highlights that specificity in dialogue is multi-faceted and that word frequency is an insufficient proxy for interesting conversation. By using linguistically-driven metrics and a "nonsense detector" trained on synthetic errors, models can generate more engaging content while maintaining logical consistency.

However, the authors note that the system still lacks a robust "world model." While the reranker catches many errors, the model does not truly "understand" the facts it generates. Future research directions include dynamic specificity—adjusting the level of detail based on conversational context—and the integration of external knowledge bases to ensure factual accuracy alongside linguistic plausibility.

Summary updated

BibTeX
@inproceedings{ko2019linguistically,
  author = {Wei-Jen Ko and Greg Durrett and Junyi Jessy Li},
  title = {{Linguistically-Informed Specificity and Semantic Plausibility for Dialogue Generation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=Sk44SQZOWH},
}
Figure