Asking the Right Question: Inferring Advice-Seeking Intentions from Personal Narratives

Liye Fu, Jonathan P. Chang, Cristian Danescu-Niculescu-MizilNAACL 2019 · DOI

When we share stories, we often have an underlying motive: we are looking for advice. However, for an AI to be truly helpful, it must look beyond the literal events of a story to understand the storyteller's intent. In their NAACL-HLT 2019 paper, Fu et al. introduce a new NLP task focused on inferring these "extradiegetic" intentions from personal narratives.

The Intention Gap in Narrative Understanding

Traditional NLP tasks often focus on intradiegetic elements—the "who, what, and where" within the story. While a model might successfully summarize a story about a sleepless night, it may fail to grasp the extradiegetic goal: why is the narrator telling us this? Are they asking for medical advice, or just wondering how to stay awake for a class?

The researchers define this as the "intention gap." To bridge it, they formulate a task where a system must identify the specific advice-seeking question a narrator intended to ask after sharing a story. This differs from standard question generation or summarization by focusing on the narrator's external goal.

Comparison of the advice-seeking intention task against existing narrative understanding benchmarks.

The ASQ Dataset and Methodology

To benchmark this capability, the authors created the ASQ (Advice-Seeking Questions) dataset. Sourced from over 400,000 posts on the Reddit community r/Advice, the final dataset consists of 21,865 instances. Each instance pairs a personal narrative with two plausible questions: the original question asked by the author and a "distractor" question from a topically similar but contextually different story.

The challenge lies in the distractor generation. The authors used a pipeline involving tf-idf similarity and symmetric selection to ensure distractors are relevant enough to be non-trivial but ultimately incorrect based on the narrative's specific cues. This ensures that the task requires more than simple keyword matching.

Experimental Results

The task was evaluated as a binary choice cloze test. While humans achieve a high accuracy of 90%, demonstrating strong social intuition, current state-of-the-art models struggle to match this performance.

Comparison of accuracy on the held-out test set between a similarity-based baseline and a finetuned language model.

As shown in the results, a finetuned Transformer-based language model reached 78.7% accuracy, outperforming a similarity-based baseline (73.4%) but still leaving a significant gap compared to human performance.

Key Insights

The study reveals that "reading between the lines" is a major hurdle for AI. In over half of the dataset, there are no explicit factual contradictions between the story and the wrong question; the correct choice requires sensing the narrator's mindset. Furthermore, models perform significantly better on specific, concrete questions than on vague ones, suggesting that AI currently relies more on explicit cues than deep intentional inference.

Future Directions

The authors suggest that future work should move from discriminative (multiple choice) to generative tasks, where models must formulate the advice-seeking question from scratch. Additionally, expanding this framework to recognize other social goals—such as seeking empathy or catharsis—is a vital step toward building more socially intelligent and empathetic AI systems.

Summary updated

BibTeX
@inproceedings{fu2019asking,
  author = {Liye Fu and Jonathan P. Chang and Cristian Danescu-Niculescu-Mizil},
  title = {{Asking the Right Question: Inferring Advice-Seeking Intentions from Personal Narratives}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/n19-1052},
}
Figure