Abstract
Modern dialogue systems often function as "literalists," excelling at surface-level text processing but failing to grasp the pragmatics of human conversation. To address this, we present DIRECT, a large-scale corpus of 71,498 pragmatic paraphrases—pairs of direct and indirect responses mapped to original dialogue contexts. Our analysis shows that indirect speech is semantically more distant and linguistically more complex than direct equivalents. Experimental results demonstrate a significant "recognition gap," where models trained on standard paraphrase datasets fail to identify pragmatic intent, highlighting the need for socially intelligent conversational agents.
Introduction: The Challenge of Pragmatics
In natural human interaction, speakers rarely state their intentions with absolute literalness. For example, if an operator asks a user if they would like to make a restaurant reservation, a response like "I don't have enough budget" implicitly carries the direct meaning: "No, please don't make a reservation. Are there any cheaper ones?"
Current NLP models are typically trained on datasets like MRPC or PAWS, which focus on lexical or syntactic variations (e.g., active vs. passive voice). These models struggle with pragmatics—the study of how context contributes to meaning. When an AI encounters indirect speech, it often fails to "read between the lines," leading to breakdowns in task-oriented dialogues.
The DIRECT Corpus
The researchers constructed the DIRECT corpus by augmenting the MultiWoZ 2.1 dataset. Using Amazon Mechanical Turk, they collected two types of paraphrases for original user utterances:
- Type-1 (Direct): An explicit, blunt version of the intent.
- Type-2 (Indirect): A natural, "roundabout" way of conveying the same message.
To ensure high quality, the team implemented automated filters to reject overly similar rephrasings and employed a second group of workers to validate that the original intent was preserved across both types.
Linguistic and Semantic Analysis
Analysis of the resulting 71,498 triples revealed several key characteristics of pragmatic speech:
- Semantic Distance: Cosine similarity between direct and indirect pairs is significantly lower than in standard paraphrase datasets. Pragmatic paraphrases are semantically "further apart," making them harder for models to link.
- Complexity: Indirect responses tend to be longer and utilize a broader vocabulary. Humans exercise more creativity and linguistic variety when being indirect, often to maintain politeness or provide context for a refusal.
- Contextual Dependency: The meaning of indirect speech is often impossible to decipher without dialogue history. For instance, "I'm still waiting on my paycheck" only functions as a "No" in the context of a purchase or booking request.
Experimental Results
The researchers established baselines using BERT for directness prediction and BART for intent transfer.
The Recognition Gap
One of the most striking findings was the failure of existing models to recognize pragmatic paraphrases. Models trained on literal datasets like MRPC showed a massive drop in recall when tested on DIRECT.
As shown in the figure above, while models achieve ~90% recall on datasets like PAWS, recall on DIRECT falls to approximately 35.3%. This confirms that pragmatic intent is a distinct and underserved challenge in current NLP.
Transfer and Prediction
- Indirect-to-Direct: Using BART with dialogue history, the model achieved a BLEU score of 33.77. Performance significantly degraded without history, proving that context is essential for decoding intent.
- Directness Ranking: A BERT-based model achieved 81.6% accuracy in ranking statements from "veiled" to "blunt," suggesting that models can be trained to detect the degree of indirectness even if they struggle to generate it.
Conclusion
The DIRECT corpus provides a vital benchmark for moving beyond literal AI. While simplifying indirect speech into direct intent is feasible with context, generating natural indirect responses remains a significant hurdle. Future work will focus on integrating DIRECT into end-to-end systems to improve the social intelligence and task success of conversational agents.
Summary updated