Exploring Conversational Adaptability: Assessing the Proficiency of Large Language Models in Dynamic Alignment with Updated User Intent

Yu-Chuan Chen, Hen-Hsen HuangAAAI 2025 · DOI

TL;DR

This paper introduces a new challenge for large language models (LLMs): adapting to user intent changes mid-conversation, a common occurrence in real-world applications like booking systems. They propose a novel dataset and evaluation framework to measure this "conversational adaptability," finding that current LLMs, including GPT-4, still struggle with dynamic alignment, with GPT-3.5 surprisingly outperforming GPT-4 on this specific task.

What Problem Does This Paper Address?

Imagine you're booking a train ticket online, and your plans change halfway through the conversation. You might initially want to go to City A but then realize City B is a better option. A good conversational AI should seamlessly pick up on this shift, discard the old information, and work with your new preference. This is precisely the problem the authors tackle: the difficulty for dialogue systems, especially LLMs, to dynamically update their understanding and adapt their responses when user intentions change during a conversation. While LLMs have made strides in understanding context, they often struggle to reconcile conflicting or outdated information from earlier turns, leading to nonsensical or unhelpful responses. This is particularly crucial in scenarios like travel booking, where plans are fluid.

Methods

The core of the paper's contribution lies in its novel approach to evaluating LLMs on this dynamic alignment task.

  • DynDST Dataset: They created a specialized dataset called DynDST, derived from the MultiWOZ 2.2 dataset. The process involved identifying user intents (represented as "facts" or slot-value pairs), intentionally introducing errors or "false turns," and then creating "update turns" where the user corrects their previous intent. This allows for testing how well LLMs can handle both implicit and explicit corrections.
  • Chain-of-Thought-Free Evaluation Framework: To address the tendency of LLMs to generate verbose and sometimes irrelevant text, the authors developed a multi-step "exact match" framework. This pipeline aims to distill the essential user intent from an LLM's response without relying on complex prompt engineering or Chain-of-Thought reasoning. It involves several stages of exact matching, along with normalization steps like lowercasing, punctuation removal, and stop-word elimination, to robustly assess alignment with the updated user intent.

Datasets

  • MultiWOZ 2.2: This existing dataset served as the foundation for creating the new DynDST dataset.
  • DynDST (Ours): A newly introduced dataset specifically designed to evaluate conversational adaptability. It contains 8,001 examples, focusing on scenarios where users modify their requests mid-conversation. It's structured to test implicit and explicit intent updates and different input lengths.

Key Insights and Ideas

  • The "Update Turn" is Key: The paper highlights that simply understanding the current utterance isn't enough. The model needs to recognize that a previous intent has been superseded and actively discard or re-contextualize outdated information.
  • GPT-3.5 Outperforms GPT-4: Perhaps the most surprising finding is that GPT-3.5 consistently outperformed GPT-4 on this specific task of dynamic alignment. The authors hypothesize this might be due to GPT-4's stronger training on safety and robustness, which could inadvertently hinder its ability to adapt to rapidly changing user intent.
  • Explicit Corrections Help: Explicit user corrections (e.g., "Should have been park, not college") combined with a "mock update" in the bot's response (e.g., "No problem at all! I have updated my memory...") significantly boost LLM performance in aligning with the new intent.
  • Naturalness Matters: The most natural conversational settings (shorter inputs, no explicit options provided in the prompt) led to better performance for models like Vicuna and Llama-2, suggesting that simpler, more human-like interactions can be more effective for these LLMs.

Key Results

  • Significant Improvement with Explicit Updates: Experiments showed that using explicit user corrections and bot acknowledgments (Exp. 2) led to a substantial boost in "Align" (the metric for matching the updated intent) compared to implicit updates (Exp. 1). For instance, GPT-3.5 saw an improvement from around 72.6% to 80.1% in its default setting.
  • GPT-3.5 vs. GPT-4: In the default settings for Exp. 1 and Exp. 2, GPT-3.5 achieved higher "Align" scores (72.6% and 80.1% respectively) than GPT-4 (66.5% and 76.0% respectively).
  • Robustness vs. Adaptability: The paper suggests a potential trade-off between robustness (as seen in GPT-4) and adaptability. GPT-4's enhanced robustness might make it less sensitive to subtle shifts in user intent.

Limitations and Future Work

  • LLM Limitations: The study clearly demonstrates that current LLMs, even state-of-the-art ones, are not perfectly aligned with dynamic user intent. They often fail to capture nuances in long-term conversations, especially in zero-shot settings.
  • Dataset Specificity: While DynDST is a valuable contribution, it's derived from MultiWOZ 2.2, which primarily focuses on task-oriented dialogues. Future work could explore this challenge in more open-domain conversational settings.
  • Future Directions: The authors suggest using these insights to further improve LLM chatbots, potentially by employing techniques like Direct Preference Optimization (DPO) to build better-aligned models. They also plan to explore the impact of different conversational settings and LLM sizes more deeply.

Summary updated

BibTeX
@inproceedings{chen2025exploring,
  author = {Yu-Chuan Chen and Hen-Hsen Huang},
  title = {{Exploring Conversational Adaptability: Assessing the Proficiency of Large Language Models in Dynamic Alignment with Updated User Intent}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i22.34534},
}
Figure