Studying Rhetorically Ambiguous Questions

Oghenevovwe Ikumariegbe, Eduardo Blanco, Ellen RiloffEMNLP 2025

TL;DR

This paper introduces SRAQ, a novel dataset of rhetorically ambiguous questions designed to challenge language models. They demonstrate that even state-of-the-art LLMs struggle to distinguish between rhetorical and informational questions when context is crucial, highlighting a significant gap in current NLP capabilities.

What Problem Does This Paper Address?

In everyday conversations, not all questions are seeking an answer. Some, known as rhetorical questions (RQs), serve other purposes like expressing emotion or emphasis. Distinguishing these from genuine informational questions (IQs) is vital for conversational AI and question-answering systems to respond appropriately. However, many RQs look identical to IQs on the surface, making them "rhetorically ambiguous." Existing datasets often lack these ambiguous cases, focusing instead on clearly distinguishable RQs. This paper aims to bridge this gap by creating a dataset that captures this crucial ambiguity and evaluating how well current language models can handle it.

Methods

The core of this work is the creation of the SRAQ dataset. The authors started with a large corpus of Reddit conversations and identified common question endings that often signal ambiguity (e.g., "Why?", "What?", "Why not?"). They then conducted a pilot study to identify the most frequent ambiguous questions and subsequently annotated a subset of these. The annotation process involved two graduate students labeling questions as either rhetorical or informational, with disagreements resolved through adjudication. This resulted in a dataset where the same question can be interpreted differently depending on the surrounding context. The authors then used this dataset to evaluate various language models, including fine-tuning and prompting approaches.

Datasets

  • SRAQ (Studying Rhetorically Ambiguous Questions): This is the novel dataset introduced in the paper. It contains rhetorically ambiguous questions sourced from Reddit, paired with contexts that allow for both rhetorical and informational interpretations.
  • SWDA-Q: Used for comparison, this dataset focuses on spoken dialogue and contains rhetorical questions.
  • QT30: Another dataset used for comparison, derived from a UK talk show, containing rhetorical questions.
  • reddit-corpus-small: A dataset from Convokit used as a starting point for sourcing Reddit conversations.

Key Insights and Ideas

  • Context is King (for Rhetorical Questions): The paper strongly emphasizes that distinguishing rhetorically ambiguous questions from informational ones is heavily dependent on context. Models that only look at the question itself or very limited context perform poorly.
  • LLMs Still Struggle with Nuance: Despite advancements, even powerful LLMs like GPT-40 and others show significant difficulty in correctly classifying rhetorically ambiguous questions, often misinterpreting them.
  • Rhetorical Questions are More Often Misclassified as Informational: The study found a consistent trend where models tend to label rhetorical questions as informational, rather than the other way around.
  • Specific Question Endings are More Ambiguous: Certain question endings, like "Why not?", exhibit higher rhetorical ambiguity, posing a greater challenge for models.
  • Fine-tuning on SRAQ is Beneficial: Fine-tuning LLMs specifically on the SRAQ dataset leads to improved performance, demonstrating the value of targeted training data for this task.

Key Results

  • SRAQ Dataset: Contains 971 instances, with 63% of questions being rhetorically ambiguous.
  • Model Performance:
    • Prompting LLMs generally outperformed baselines.
  • Fine-tuning LLMs on SRAQ showed significant improvements, with fine-tuned GPT-40 achieving macro F1 scores up to 0.84.
  • Baselines trained on other datasets (SWDA-Q, QT30) performed worse on SRAQ, highlighting the dataset's unique challenge.
  • The best-performing fine-tuned model (GPT-40) achieved an F1 score of 0.84 on "So" questions but struggled with "Why not?" questions, scoring 0.70.
  • Error Analysis: Misclassifications often occurred with negative assertions, shortened contexts, and missed subtle cues like repeated punctuation. Clarification-seeking IQs were sometimes mistaken for RQs.

Limitations and Future Work

  • Dataset Size: The SRAQ dataset, while valuable, is limited in size due to the human effort involved in annotation. The authors suggest future work could expand the set of question endings and scale up annotation efforts.
  • Closed-Source Models: The use of closed-source models for some experiments incurred costs and limits full transparency.
  • Ethical Considerations: The dataset, sourced from Reddit, may contain offensive language or biases. Users are encouraged to be mindful of this.
  • Future Work: The authors hope SRAQ will encourage more research into rhetorically ambiguous questions and inspire further development of models that can better understand this nuanced aspect of language. They also suggest exploring more context settings and a wider range of question endings.

Summary updated

BibTeX
@inproceedings{ikumariegbe2025studying,
  author = {Oghenevovwe Ikumariegbe and Eduardo Blanco and Ellen Riloff},
  title = {{Studying Rhetorically Ambiguous Questions}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure