From Chat Logs to Collective Insights: Aggregative Question Answering

Wentao Zhang, Woojeong Kim, Yuntian Deng

TL;DR

This paper introduces "Aggregative Question Answering" (AQA), a new task that aims to extract collective insights from massive conversational datasets generated by LLM-powered chatbots. They propose the WildChat-AQA benchmark, a challenging dataset of over 6,000 questions derived from 180,000 conversations, and demonstrate that current state-of-the-art methods struggle to perform this task effectively due to computational costs and reasoning limitations.

What Problem Does This Paper Address?

Conversational AI, powered by LLMs, is generating unprecedented volumes of data. While this data offers a rich lens into societal interests, trending topics, and collective concerns, current research largely treats these interactions as independent. This overlooks critical insights that could emerge from aggregating and reasoning across large-scale conversation logs. Existing approaches typically focus on improving individual chatbot responses or summarizing single documents. The paper argues that this "i.i.d." assumption misses valuable contextual features (temporal, geographical, device-related) that are crucial for understanding regional differences, temporal shifts in attitudes, and demographic-specific concerns. The core challenge is to move beyond individual conversation analysis to global-scale reasoning over thousands of interactions to answer aggregative queries.

Methods

The paper introduces a novel task, Aggregative Question Answering (AQA), which requires models to reason over large collections of user-chatbot interactions to answer questions about collective insights. To support this task, they developed the WildChat-AQA benchmark.

A key methodological contribution is the PROBE (Probing Retrieval Of Broad Evidence) approach. Unlike traditional retrieval methods that focus on a small set of highly specific documents, PROBE aims to identify a broader range of documents to support aggregative reasoning. It works in two steps:

  1. Broad Query Generation: An LLM generates a comprehensive set of short, diverse queries related to the original question, along with strict filtering conditions to exclude irrelevant documents.
  2. Evidence Aggregation and Generation: Each generated query and its filters are used individually to retrieve relevant documents. These results are then aggregated by merging the retrieved lists and using max pooling to assign the highest relevance score. The top k documents from this aggregated list serve as supporting evidence.

Datasets

The paper introduces the WildChat-AQA benchmark, a new dataset specifically designed for Aggregative Question Answering.

  • Source: Derived from the WildChat dataset, which contains over 182,330 real-world user-chatbot conversations.
  • Content: Includes conversation transcripts and metadata such as temporal, geographical, and user-specific information.
  • New Contributions: WildChat-AQA formulates 6,027 aggregative questions that require reasoning over explicit and implicit attributes of conversations (topics, keywords, locations, time).
  • Creation Process: The dataset was constructed through a multi-step process involving preprocessing, topic discovery (using GPT-4o and TnT-LLM), keyword categorization, and question generation.

Key Insights and Ideas

  • The "i.i.d." assumption is a bottleneck: Treating conversations as independent data points prevents the extraction of valuable collective insights that arise from temporal patterns and thematic connections.
  • Aggregative reasoning is distinct from summarization: AQA requires global-scale reasoning conditioned on the query, not just condensing information from a few documents.
  • PROBE is crucial for broad evidence retrieval: Traditional RAG methods are insufficient; PROBE's ability to generate diverse queries and filter effectively is key to gathering the broad evidence needed for aggregative insights.
  • Summarized conversations are more effective: Models perform better with summarized inputs, highlighting the challenge of aggregating information from long, noisy raw texts.
  • Performance scales with context: Unlike standard RAG, AQA fundamentally relies on a broader set of documents, and performance significantly improves as more documents are provided.
  • Explicit reasoning ("think" mode) boosts performance: Models that incorporate explicit reasoning steps show significant improvements, underscoring the reasoning-intensive nature of AQA.
  • Token consumption is a major challenge: Achieving good performance requires processing very large input tokens, posing a significant computational hurdle.

Key Results

  • Baselines are ineffective: Simple textual relevance models (BM25, Cosine Sim) performed no better than random selection.
  • PROBE outperforms standard RAG: PROBE consistently showed large performance improvements across various models and conversation formats, with NDCG@1 scores improving by up to 24.7 points.
  • Summaries are better than raw conversations: Summarized inputs led to improved NDCG@1 scores of 4.5 to 14.4 points over raw conversations.
  • Stronger models yield better results: o4-mini consistently achieved the highest performance, with a maximum NDCG@1 score of 0.7571. Qwen3-32B-think was the top-performing open-source model.
  • Basic finetuning is insufficient: Direct finetuning on raw or summarized conversations without explicit aggregative reasoning steps did not significantly exceed random performance.
  • Retrieval effectiveness is crucial: Higher recall rates consistently led to better NDCG scores.
  • More context improves performance: Performance significantly improves as more documents are provided, validating the need for extensive context and global dataset knowledge.
  • Reasoning improves performance: Models using "think" mode showed significant performance gains, indicating the importance of explicit reasoning.

Limitations and Future Work

  • Model-derived annotations: While LLMs were used for attribute inference, errors and inconsistencies can occur due to model hallucinations or instruction misalignment. Human evaluations confirmed moderate to high accuracy but acknowledged that some errors are inevitable.
  • Artificiality of generated questions: Questions were generated by GPT-4.1, which might introduce stylistic artifacts. Strong performance on WildChat-AQA may not directly generalize to human-authored questions.
  • Computational cost: Processing extremely large amounts of text for AQA leads to substantial computational costs.
  • Reasoning over very long context: Current models have limited context windows, and performance degrades with longer inputs. Developing efficient methods for reasoning over very long contexts is an open problem.
  • Cost-efficient AQA: Future research could explore hierarchical indexing, retrieval strategies, and long-term memory mechanisms to reduce token consumption and improve efficiency.
  • Streaming AQA: Developing methods to dynamically update aggregational insights as conversations arrive in streams is a promising direction for real-time analysis.
  • Ethical considerations: The paper acknowledges potential ethical concerns, especially when insights relate to sensitive topics. They avoided targeting protected attributes and used anonymized data, aiming to foster transparent research.

Summary updated

BibTeX
@inproceedings{zhang2025from,
  author = {Wentao Zhang and Woojeong Kim and Yuntian Deng},
  title = {{From Chat Logs to Collective Insights: Aggregative Question Answering}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure