From Chat Logs to Collective Insights: Aggregative Question Answering

Wentao Zhang, Woojeong Kim, Yuntian Deng

Abstract

Traditional question-answering systems focus on retrieving specific facts from individual documents. However, as large-scale chatbot interactions proliferate, there is a growing need to extract "big picture" insights from thousands of conversations. This paper introduces Aggregative Question Answering (AQA), a task requiring models to synthesize collective intelligence from massive datasets. To facilitate research, the authors present WildChat-AQA, a benchmark of over 6,000 questions derived from real-world logs, and PROBE, a retrieval method designed to capture a broad spectrum of evidence. Experimental results demonstrate that specialized retrieval and reasoning-intensive models are essential for mastering this complex task.

Introduction: Beyond the Needle in the Haystack

Current NLP research often treats chatbot interactions as isolated data points, primarily using them to fine-tune model helpfulness. This approach overlooks the potential of these logs as a source of collective intelligence. Aggregating thousands of conversations can reveal emerging societal trends, regional concerns, or shifts in public sentiment—answering questions like, "What are the most popular topics among users in California this week?"

Conceptual overview of Aggregative Question Answering (AQA) compared to standard summarization and database aggregation.

Existing techniques like standard Retrieval-Augmented Generation (RAG) are designed to find a "needle in a haystack." In contrast, AQA requires characterizing the entire haystack. Standard models struggle with this because they often fail to retrieve a diverse enough set of documents or cannot process the computational load of thousands of noisy, long-form chat logs.

The WildChat-AQA Benchmark

To address these challenges, the authors developed WildChat-AQA, a benchmark derived from the WildChat dataset containing over 1 million real-world ChatGPT logs. The benchmark consists of 6,027 aggregative questions linked to 182,330 conversations. These questions incorporate metadata such as timestamps and locations, alongside inferred attributes like topics and keywords.

The creation of WildChat-AQA involved a sophisticated four-step pipeline:

  1. Preprocessing: Deduplication and filtering of noisy or irrelevant logs.
  2. Topic Discovery: Using TnT-LLM for recursive clustering into a hierarchy of 28 high-level and 455 fine-grained topics.
  3. Keyword Categorization: Detecting and aggregating keywords into broader categories.
  4. Question Generation: Converting structured database queries into natural language questions using GPT-4.1.
The four-step pipeline used to transform raw chat logs into the structured WildChat-AQA benchmark.

Methods: PROBE and Summarization

The paper introduces PROBE (Probing Retrieval Of Broad Evidence) to overcome the limitations of standard RAG. While standard RAG uses a single query to find similar documents, PROBE prompts an LLM to generate a set of diverse search queries and strict filters. This ensures the retrieval of a representative sample of evidence across different facets of a topic, capturing the "long tail" of data necessary for accurate aggregation.

Furthermore, the study highlights the power of summarization. Models perform significantly better when reasoning over summarized conversations (averaging 21 tokens) compared to raw logs (averaging 1,143 tokens). Summaries effectively remove noise, allowing models to process a much larger volume of data within their context windows.

Key Results and Insights

The evaluation of various models on WildChat-AQA yielded several critical insights:

  • PROBE Dominance: PROBE consistently outperformed standard RAG, improving NDCG@1 scores by 14.8 to 24.7 points across different models when using raw data.
  • Reasoning is Essential: Models with explicit reasoning steps, such as OpenAI’s o4-mini, achieved the highest performance (NDCG@1 of 0.7571). This suggests that AQA is not merely a pattern-matching task but requires high-level synthesis.
  • Failure of Simple Baselines: Traditional methods like BM25 and Cosine Similarity performed no better than random selection, confirming that keyword matching is insufficient for aggregative reasoning.
  • Scaling with Context: Performance improves as the number of provided conversations increases, validating the necessity of global dataset knowledge for this task.

Conclusion and Future Directions

Aggregative Question Answering represents a shift from analyzing individual interactions to understanding collective patterns. While the current work establishes a strong foundation, challenges remain regarding the computational cost of processing thousands of inputs. Future research may explore Streaming AQA, enabling models to update insights in real-time as new data arrives. Additionally, the authors emphasize the importance of privacy and ethics when aggregating user data, advocating for responsible deployment that avoids demographic profiling.

Summary updated

BibTeX
@inproceedings{zhang2025from,
  author = {Wentao Zhang and Woojeong Kim and Yuntian Deng},
  title = {{From Chat Logs to Collective Insights: Aggregative Question Answering}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure