Put Chatbot into Its Interlocutor's Shoes: New Framework to Learn Chatbot Responding with Intention

Hsuan Su, Jiun-Hao Jhan, Fan-Yun Sun, Saurav Sahay, Hung-yi Lee

Abstract: Hsuan Su, Jiun-Hao Jhan, Fan-yun Sun, Saurav Sahay, Hung-yi Lee. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

2021.

Introduction

Modern conversational agents have achieved remarkable levels of fluency and coherence. However, most existing models are fundamentally reactive; they aim to provide a relevant response to the immediate preceding turn without a broader conversational objective. In contrast, human communication is inherently proactive and intentional. When we engage in dialogue, we often have specific goals—whether it is to cheer up a distressed friend, steer a conversation toward a particular topic, or encourage a reticent speaker to share more.

A comparison between a typical reactive chatbot and the proposed guiding chatbot, which aims to steer the interlocutor toward a target emotion.

This paper addresses the limitation of reactive models by introducing a framework that allows chatbots to understand and influence the effect their utterances have on an interlocutor. By putting the chatbot "into its interlocutor's shoes," the model learns to generate responses that achieve specific conversational intentions while maintaining natural dialogue flow.

Methodology: The Guiding Chatbot Framework

The researchers propose a reinforcement learning (RL) framework centered on two primary agents: a Guiding Chatbot (C) and an Interlocutor Model (I).

The interaction loop of the proposed framework, where the Guiding Chatbot's response is evaluated by a Reward Function based on the Interlocutor Model's subsequent reaction.

The Interaction Loop

The core of the framework is an interaction loop where the Guiding Chatbot generates a response $C(x)$ to an input $x$. This response is then processed by the Interlocutor Model, which represents either a human user or another automated agent. The Interlocutor’s subsequent reply $I(C(x))$ serves as the "environment" from which a reward is derived.

Reinforcement Learning and Reward Design

The Guiding Chatbot is trained using policy gradients to maximize a total reward ($R$) that balances intention with coherence:

  1. Intention Reward ($R_I$): This component measures how effectively the interlocutor's response aligns with the desired goal. The authors explored three primary dimensions:
  2. Sentence Length ($R_L$): Aimed at increasing the verbosity of the listener.
  3. Emotion ($R_E$): Aimed at inducing specific moods (e.g., joy, anxiety), quantified using a BERT-based Emotion Detector in a 2D Valence-Arousal space.
  4. Specific Words ($R_W$): Aimed at prompting the listener to use vocabulary from predefined categories like "Food" or "Sports."
  5. Coherence Reward ($R_C$): To ensure the chatbot remains a viable conversational partner, a fixed GPT-2 model provides a reward based on the probability and relevance of the chatbot's output. This prevents the model from "gaming" the intention reward with nonsensical but effective trigger phrases.

The Guiding Chatbot was implemented using DialoGPT and tested against various interlocutor models, including retrieval-based systems and Google’s chatbot.

Experimental Setup

The models were trained and evaluated using several key resources:

  • EmpatheticDialogues (ED): This dataset, containing 25,000 conversations grounded in emotional contexts, was used to fine-tune base models and train the Emotion Detector.
  • Targeted Word Lists: Categories for the "Specific Words" task were drawn from Google’s "Bad Words" lists and educational resources for "Sports" and "Food."
  • Cornell Movie Dialogue: This dataset served as a benchmark for analyzing the inherent biases and linguistic tendencies of different models during the discussion phase.

Results and Key Insights

The experimental results demonstrated that the framework successfully enables intentional communication:

  • Inducing Engagement: The guiding chatbot successfully increased the average response length of a Google bot interlocutor from 3.74 to 10.14 words.
  • Emotional Steering: The framework significantly reduced the Mean Square Error (MSE) between target emotions and actual interlocutor responses across all tested models.
  • Human Validation: In trials with 19 human participants, the chatbot successfully induced target emotions and specific word usage. However, humans noted a "relevance trade-off": as the drive toward a specific intention increased, the chatbot's responses occasionally felt less connected to the immediate conversational context.

A significant insight from this work is the use of the interlocutor as the RL environment. This shift allows the chatbot to learn how to "influence" rather than just "talk." Furthermore, the researchers found that training against multiple diverse interlocutor models simultaneously improved the bot's ability to generalize when interacting with real humans.

Discussion and Ethical Considerations

While the ability to steer conversations offers benefits for applications like mental health support or educational tutoring, the authors highlight significant ethical risks. Such technology could potentially be exploited for emotional manipulation, social antagonism, or political interference.

To mitigate these risks, the authors suggest the development of "Intention Classifiers" to detect manipulative behavior in AI and the use of "Special Tokens" to transparently label AI-generated speech that has a specific underlying intention. Future research will focus on better balancing the tension between achieving conversational goals and maintaining high contextual relevance.

Summary updated

BibTeX
@inproceedings{su2021put,
  author = {Hsuan Su and Jiun-Hao Jhan and Fan-Yun Sun and Saurav Sahay and Hung-yi Lee},
  title = {{Put Chatbot into Its Interlocutor's Shoes: New Framework to Learn Chatbot Responding with Intention}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=H7ArhsUAev},
}
Figure