FQ-Eval: Building Evaluation Dataset for User-centered Follow-up Question Generation

Sanghyun Seo, Bumsoo Kang, Dahm Lee, Jaeheon Kim, Joongbo Shin, Eui Soon Kim, Kijeong JeonEMNLP 2025

TL;DR

This paper introduces FQ-Eval, a novel dataset and evaluation framework for assessing how well AI chatbots generate user-centered follow-up questions. Unlike previous work focusing on relevance, FQ-Eval captures crucial human-aligned criteria like "Creative Leap" and "LLM Enablement," offering a more holistic way to evaluate and improve conversational AI.

What Problem Does This Paper Address?

As AI chatbots become integrated into our daily lives, users often struggle to articulate their needs and goals effectively. While chatbots excel at answering direct questions, they can falter when conversations become ambiguous or when users need help exploring topics or leveraging the AI's full potential. Existing research on follow-up question generation has primarily focused on information retrieval and topical relevance, often overlooking how these questions can truly support users' intrinsic needs and conversational goals. This gap can lead to misaligned questions and decreased user satisfaction, especially for novice users. FQ-Eval aims to bridge this by providing a user-centered evaluation standard that goes beyond simple relevance.

Methods

The core of this work is the creation of the FQ-Eval dataset and its associated evaluation methodology.

  1. Human-Aligned Criteria Development: The authors conducted a multi-stage user study involving semi-structured interviews, focus groups, and affinity diagramming. This process distilled user expectations into five key criteria:
  2. Exploratory Scope: Broadening and deepening the conversation.
  3. Contextual Relevance: Maintaining logical flow and topic continuity.
  4. Creative Leap: Inspiring new ideas and unique perspectives.
  5. LLM Enablement: Guiding users to leverage the AI's capabilities effectively.
  6. Guided Onboarding: Helping users start exploring new topics or complex concepts.
  7. Dataset Construction:
    • Seed Question Generation: Real-world LLM use cases from a Harvard Business Review study were adapted. GPT-4.1 was used to generate candidate questions, which were then refined by human annotators to ensure realism and alignment with use cases. This resulted in 200 diverse question-answer (QA) pairs.
  8. Follow-up Question Curation: For each QA pair, GPT-4.1 generated eight candidate follow-up questions per criterion. Human annotators then selected and refined these candidates to create high-quality, criterion-aligned follow-up questions. This process yielded 1,000 follow-up questions, forming the FQ-Eval dataset.
  9. Evaluation Methodologies:
    • n-way Selection: Evaluators (human or LLM-judge) choose the best follow-up question from a set of candidates based on a specific criterion.
  10. Score-based Evaluation: Follow-up questions are scored individually on a 1-5 scale against each criterion.

Datasets

  • FQ-Eval: The primary contribution is this newly introduced dataset. It comprises 200 carefully curated single-turn QA pairs and 1,000 diverse follow-up questions, grounded in practical generative AI use cases and aligned with five human-derived criteria. It aims to be user-centered and scenario-driven.
  • Existing Datasets (for comparison):
    • FQ-Bank: A large, multi-turn dataset with 662 dialogues and over 2,000 QA pairs from the OrQuAC corpus.
  • FollowupQG: A single-turn dataset with 501 QA pairs from Reddit social QA threads.

Key Insights and Ideas

  • Beyond Relevance: The study strongly suggests that user satisfaction with AI conversations hinges on more than just topical relevance. Criteria like "Creative Leap" and "LLM Enablement" are crucial for deeper engagement and effective AI utilization, especially for novice users.
  • LLM Judges are Robust: The research demonstrates that LLM judges (like GPT-4.1) can provide highly consistent and reliable evaluations, correlating well with human judgments, making them a viable tool for large-scale assessment.
  • Model Capabilities Vary by Criterion: While LLMs are improving, they still struggle with criteria like "Creative Leap" and "Guided Onboarding," indicating areas ripe for future research and development.
  • FQ-Eval's Practicality: The dataset and evaluation framework are designed to be practical, enabling service providers to diagnose specific areas for improvement in their AI's follow-up question generation.

Key Results

  • FQ-Eval Dominance: In n-way selection tasks, FQ-Eval's representative follow-up questions consistently achieved the highest win rates and best average ranks across most criteria, demonstrating strong alignment with human preferences.
  • LLM Performance: Flagship LLMs showed strong performance in "Contextual Relevance" but lagged in "Creative Leap" and "LLM Enablement." Smaller models sometimes outperformed larger ones on specific criteria.
  • Real-world Service Assessment: Using Perplexity as an example, FQ-Eval's scoring-based evaluation revealed that while the service excels at information retrieval, its follow-up questions are surface-level for advanced aspects like "LLM Enablement."
  • LLM Judge Consistency: Inter-model agreement among LLM judges was high, with adjacent agreement reaching 97.2%, indicating reliable evaluation outcomes.

Limitations and Future Work

  • Broader Use Cases: The current dataset, while diverse, might not capture all real-world scenarios. Future work could explore use cases across different user demographics (age, gender, occupation) and adapt questions for less experienced users. Cultural biases in criteria applicability also need examination.
  • Fact-Checking: The current evaluation does not verify answer correctness. Future work could extend the framework to assess how follow-up questions adapt to flawed or misleading answers.
  • Dataset Size: While sufficient for initial evaluation, the dataset size (200 QA pairs, 1,000 follow-ups) could be expanded for greater generalizability.
  • Multi-turn Conversations: FQ-Eval currently focuses on single-turn interactions. Future research should extend evaluation to more complex, multi-turn dialogues, potentially incorporating techniques like Retrieval-Augmented Generation (RAG).

Summary updated

BibTeX
@inproceedings{seo2025fq,
  author = {Sanghyun Seo and Bumsoo Kang and Dahm Lee and Jaeheon Kim and Joongbo Shin and Eui Soon Kim and Kijeong Jeon},
  title = {{FQ-Eval: Building Evaluation Dataset for User-centered Follow-up Question Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure