What Are the Implications of Your Question? Non-Information Seeking Question-Type Identification in CNN Transcripts

Yao Sun, Anastasiia Tatlubaeva, Zhihan Li, Chester Palen-MichelCOLING 2024

TL;DR

This paper introduces the first publicly available dataset for classifying "non-information seeking questions" (NISQs) in conversational transcripts, a crucial but understudied area for improving AI's conversational abilities. They establish baselines using transformer models and find that while NISQ classification is complex, context and speaker information are vital for accurate identification.

What Problem Does This Paper Address?

Conversational AI systems are getting good at understanding and answering "information-seeking questions" (ISQs) – the kind where users explicitly ask for facts. However, human conversations are full of other types of questions, called "non-information seeking questions" (NISQs). These questions, like rhetorical questions or tag questions, don't necessarily seek an answer but serve to underscore a point, seek affirmation, or express emotion. Understanding these NISQs is key to making AI interactions more natural and nuanced. The problem is that there's a lack of dedicated datasets and research for classifying these fine-grained NISQ types, making it hard for AI to grasp the subtle dynamics of human discourse.

Methods

The researchers tackled this by creating a new dataset and then evaluating machine learning models on the task of classifying NISQ types.

  • Dataset Creation: They started with the RQueT dataset, which contains CNN transcripts labeled as either ISQ or NISQ. They then refined and consolidated the NISQ categories into seven labels: DELIBERATIVE, RHETORICAL, ECHO, TAG, QUOTED, OTHER, and ISQ. To ensure objectivity, they re-annotated the data with four graduate students specializing in computational linguistics. They also implemented an adjudication process with three expert annotators to resolve disagreements.
  • Model Evaluation: They performed multi-class classification using four final labels: DELIBERATIVE, RHETORICAL, OTHER, and ISQ. They fine-tuned two powerful transformer models: RoBERTa and BigBird. They experimented with different amounts of context provided to the models – just the target question, one sentence before and after, or two sentences before and after.

Datasets

  • RQueT (Resource of Question Types): The original source of CNN transcripts used for annotation.
  • Newly Introduced NISQ Dataset: A publicly available corpus of 1,508 annotated questions from CNN transcripts, featuring fine-grained NISQ types. This serves as an initial benchmark for NISQ classification.

Key Insights and Ideas

  • NISQ Classification is Hard: The results highlight the inherent complexity of distinguishing between different types of NISQs, even for human annotators. This suggests that AI will face significant challenges in mastering these nuances.
  • Context is King (But Speaker Info Might Be More So): Surprisingly, the model that received the least context (only the target question) performed best. This contradicts the hypothesis that more context would improve performance. The authors speculate this is because their models ignored speaker information, which previous research (Kalouli et al., 2021) found to be highly predictive for ISQ/NISQ classification.
  • LLMs as Pre-Annotators: While a fine-tuned RoBERTa model outperformed GPT-3.5 Turbo on this specific task, the LLM's ability to achieve 42% accuracy in just 2 minutes suggests it could be a valuable tool for pre-annotation, speeding up the process for human experts.

Key Results

  • Human Annotation Challenges: Inter-annotator agreement metrics (Cohen's Kappa, Bennet's S, Krippendorff's alpha) were relatively low, especially before the introduction of a superlabel, indicating the subjective nature of NISQ classification.
  • Model Performance:
    • The best-performing model (RoBERTa with only the target question) achieved an accuracy of 64% and an MCC of 0.38.
  • BigBird models with more context performed worse than the simpler RoBERTa model, with accuracies of 51% and 54%.
  • GPT-3.5 Turbo achieved 42% accuracy on a subset of 100 questions.

Limitations and Future Work

  • Dataset Bias: The current dataset is heavily skewed towards political discussions from CNN transcripts. This bias can lead to misinterpretations due to political rhetoric and requires a deep understanding of US and global politics.
  • Limited Context: The RQueT dataset only provides five sentences per conversation, which can be insufficient for annotators to confidently determine question types.
  • Speaker Information Ignored: The models did not utilize speaker information, which previous work has shown to be crucial for question classification.
  • Future Directions:
    • Diversify Dataset: Create a dataset that covers everyday, relatable conversations on broader topics to achieve a more generalized understanding.
  • Integrate Speaker Information: Incorporate speaker details into models to better understand context and differentiate between speaking styles.
  • Explore More ML Methods: Investigate further machine learning methodologies for NISQ identification.
  • LLM for Pre-Annotation: Leverage LLMs for efficient pre-annotation to streamline the annotation process.

Summary updated

BibTeX
@inproceedings{sun2024what,
  author = {Yao Sun and Anastasiia Tatlubaeva and Zhihan Li and Chester Palen-Michel},
  title = {{What Are the Implications of Your Question? Non-Information Seeking Question-Type Identification in CNN Transcripts}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}

Loading related content...

Figure