Proactive User Information Acquisition via Chats on User-Favored Topics

Shiki Sato, Jun Baba, Asahi Hentona, Shinji Iwata, Akifumi Yoshimoto, Koichiro YoshinoEMNLP 2025

TL;DR

This paper introduces "Proactive Information Acquisition" (PIA), a new task for dialogue systems that need to gather specific user information during casual conversations on user-preferred topics without feeling abrupt. They developed a dataset and a strategy-based system that outperforms standard LLMs, demonstrating the feasibility of automated evaluation for this crucial capability.

What Problem Does This Paper Address?

Modern dialogue systems are moving beyond simple chit-chat to provide tangible benefits, like health advice or news delivery. To do this effectively, systems often need to collect specific user information. However, directly asking for this information can feel abrupt and disrupt the user's experience, especially when the requested information is unrelated to the ongoing conversation topic. This paper addresses the challenge of proactively acquiring necessary user information during natural, user-favored conversations without alienating the user.

Methods

The authors propose the Proactive Information Acquisition (PIA) task, which requires a system to obtain answers to predefined questions (QUESTIONS) while maintaining a conversation on a given topic (TOPIC) that may not be directly related to the questions. They developed a strategy-based system that leverages insights from analyzing LLM failures. This system incorporates a Belief-Desire-Intention (BDI) model framework. Key components include:

  • Belief Update: The system predicts the user's answer to the QUESTION using an LLM, determining if information acquisition is complete.
  • Desire Generation: Based on the belief update, the system decides whether to continue information acquisition or focus solely on the TOPIC.
  • Response Candidate Generation: The system generates several types of response candidates:
  • Key Utterance Candidates: Based on seven predefined relationship types between the TOPIC and QUESTION, aiming to naturally introduce the question.
  • Cushion Utterance Candidates: Used to smoothly transition to a key utterance.
  • Vanilla Candidate: A standard LLM response.
  • Safe Candidate: A response focused solely on the TOPIC, used when information acquisition is complete.
  • Intention Generation: An LLM evaluator assesses the abruptness of candidate utterances, and the system selects the least abrupt, non-abrupt candidate.

Datasets

The authors constructed a dataset of 650 PIA chats. This dataset was created in two stages:

  • Pilot Collection: 200 chats involving human evaluators and LLMs.
  • Refined Collection: 450 chats using an LLM-based response generation framework to improve success rates.

The dataset includes human evaluations of abruptness and predictability for each chat, comprising 5,850 user utterances and 5,200 system utterances.

Key Insights and Ideas

  • PIA is a distinct and challenging task: Acquiring information naturally within a user-preferred topic requires more than just task instructions for LLMs.
  • Abruptness is a major failure point: LLMs frequently produce abrupt utterances when attempting PIA, often by introducing the QUESTION too suddenly or unnaturally.
  • Relationship types matter: Successful PIA chats often leverage specific relationship types (e.g., SUB-THEME, PLACE, MEANS) to connect the TOPIC and QUESTION.
  • Cushion utterances can help: A single, well-placed "cushion" utterance can significantly improve the smoothness of introducing a key question.
  • LLMs can be trained to detect abruptness: Fine-tuning LLMs with human evaluations significantly improves their ability to identify abrupt utterances.
  • Automated evaluation is feasible: The PIA task can be evaluated automatically with high accuracy, enabling faster iteration and development.

Key Results

  • The strategy-based system significantly reduced abrupt utterances and improved task success rates compared to standard LLM prompting methods.
  • In the Pilot Collection, even recent LLMs achieved success rates below 20%, with abrupt utterances occurring in over 78% of chats.
  • The fine-tuned LLM evaluator achieved an F1 score of 88.5 for detecting abruptness, demonstrating high accuracy.
  • The strategy-based system achieved a 40% success rate (SUC) in the task, a substantial improvement over baseline systems.
  • Semi-automatic and full-automatic evaluation methods showed strong correlations with human evaluations for key metrics like ACQ and N-ABR.

Limitations and Future Work

  • Artificial Persona Data: The study used artificial personas for privacy and reproducibility. Future work could explore acquiring actual user information and its impact on PIA.
  • Question Formats: The study exclusively used Yes-No questions. Future research could investigate open-ended or ranking questions.
  • Domain and User Dependency: While claimed to be domain-independent, the current evaluation doesn't deeply analyze how specific topic-question combinations or user characteristics affect performance. Future work should explore these dependencies.
  • Relationship between System and Users: The study did not define the relationship between the system and user, which could influence information acquisition strategies.
  • Cushion Utterance Drawbacks: While helpful, cushion utterances might lead to difficulties if users provide unexpected responses. Robust recovery strategies are needed.
  • Prompt Dependence: Experimental results might be influenced by the specific prompts used.

Summary updated

BibTeX
@inproceedings{sato2025proactive,
  author = {Shiki Sato and Jun Baba and Asahi Hentona and Shinji Iwata and Akifumi Yoshimoto and Koichiro Yoshino},
  title = {{Proactive User Information Acquisition via Chats on User-Favored Topics}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure