What makes a good conversation? How controllable attributes affect human judgments

Abigail See, Stephen Roller, Douwe Kiela, Jason Weston

Introduction

Building conversational agents that feel "human" and "engaging" remains a significant challenge in natural language processing. While neural models have improved at next-utterance prediction, they often struggle with the nuances of multi-turn dialogue, frequently becoming repetitive or overly generic. This research investigates how fine-tuning low-level "knobs"—such as repetition, specificity, and question-asking—impacts human judgments of conversation quality.

Conceptual framework mapping low-level controllable attributes to human judgments of conversational aspects and overall quality.

Methodology

The researchers employed two primary techniques to inject control into neural text generation:

  1. Conditional Training (CT): Models are trained with discrete control variables (e.g., specificity levels), allowing users to specify desired attributes at test time.
  2. Weighted Decoding (WD): A test-time intervention that modifies the scoring function during beam search to reward or penalize specific features, such as rare words or repeated n-grams.

The study targeted four key attributes: repetition (n-gram overlap), specificity (normalized inverse document frequency), response-relatedness (sentence embedding similarity), and question-asking (presence of "?"). Models were pre-trained on 2.5 million Twitter pairs and fine-tuned on the PersonaChat dataset.

Key Insights: The "Goldilocks" Zone and Beyond

The study revealed several critical insights into what humans value in a digital conversation partner:

  • Repetition is the "Silent Killer": Eliminating repetition is the single most effective baseline improvement. Simple n-gram blocking significantly boosted every human metric across the board.
  • Optimal Specificity: There is a non-linear relationship between specificity and engagement. While generic responses are perceived as boring, excessive specificity can lead to nonsensical content or "hallucinations" that disrupt the flow.
  • The Inquisitiveness Paradox: Humans preferred bots that asked questions approximately 65% of the time—more than double the rate found in human-to-human gold data (28.8%). Users seem to enjoy the engagement of being asked about themselves.
The relationship between specificity control and human-rated engagingness, illustrating the peak "Goldilocks" zone for detail.

Results and Performance

By systematically optimizing these controllable attributes, the authors developed a model that matched the "engagingness" of state-of-the-art systems like the NeurIPS ConvAI2 winner ("Lost in Conversation"). Notably, this was achieved using 12x less training data, highlighting that strategic generation control can be as effective as massive scale.

In A/B testing, specificity-controlled models were preferred over baselines up to 84% of the time. Furthermore, increasing specificity and response-relatedness directly improved "listening" scores, suggesting that users perceive more detailed, on-topic responses as evidence of the bot's attention.

Limitations and Future Directions

A primary trade-off exists between "interestingness" and "making sense." High-control settings, particularly in weighted decoding, can sometimes produce ungrammatical or "degenerate" text. Additionally, while this study used static control settings, future work could explore dynamic control, where the model adjusts its attributes (e.g., becoming more inquisitive or specific) based on the real-time state and sentiment of the conversation.

Summary updated

BibTeX
@inproceedings{see2019what,
  author = {Abigail See and Stephen Roller and Douwe Kiela and Jason Weston},
  title = {{What makes a good conversation? How controllable attributes affect human judgments}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=SkbgaXWuZS},
}
Figure