NaRLE: Natural Language Models using Reinforcement Learning with Emotion Feedback

Ruijie Zhou, Soham Deshmukh, Jeremiah Greer, Charles LeearXiv 2021 · OpenReview

Abstract

Current research in dialogue systems is focused on conversational assistants working on short conversations in either task-oriented or open domain settings. In this paper, we focus on improving task-based conversational assistants online, primarily those working on document-type conversations (e.g., emails) whose contents may or may not be completely related to the assistant's task. We propose "NARLE" a deep reinforcement learning (RL) framework for improving the natural language understanding (NLU) component of dialogue systems online without the need to collect human labels for customer data. The proposed solution associates user emotion with the assistant's action and uses that to improve NLU models using policy gradients. For two intent classification problems, we empirically show that using reinforcement learning to fine tune the pre-trained supervised learning models improves performance up to 43%. Furthermore, we demonstrate the robustness of the method to partial and noisy implicit feedback.

Introduction

Building effective AI assistants for "document-type" conversations, such as managing calendars or tasks via email, presents unique challenges compared to standard chit-chat bots. These interactions are typically long, contain multiple intents within a single turn, and are often buried within irrelevant contextual information. Traditional supervised learning (SL) approaches to improving these models face significant obstacles. First, manual annotation is prohibitively expensive and slow. Second, user behavior is dynamic, leading to distribution shifts that render offline-trained models obsolete. Finally, in commercial environments, privacy regulations often prevent developers from accessing actual customer data for labeling—the "no eyes-on access" constraint.

To address these issues, we introduce NaRLE (Natural Language Models using Reinforcement Learning with Emotion Feedback). NaRLE leverages the implicit emotional cues users provide in their natural responses as a weak reward signal to fine-tune NLU models online.

The NaRLE Framework

The NaRLE architecture integrates Natural Language Understanding with a Reinforcement Learning loop, allowing the agent to learn directly from user interactions.

Architecture of the NaRLE framework showing the feedback loop from user response to model update.

The framework consists of four primary components:

  1. ScopeIt Filter: Emails are often verbose. This module extracts only the sentences relevant to the task and those expressing emotion, filtering out noise.
  2. Emotion Detector: A DistilBERT-based model classifies user responses into Positive, Negative, or Neutral categories.
  3. Reward Logic: Detected emotions are mapped to numerical values (+1 for positive, -1 for negative, and 0 for neutral).
  4. Policy Gradient (REINFORCE): The system uses the REINFORCE algorithm to update the NLU model. When an action results in a positive user emotion, the model's parameters are adjusted to favor that action in similar future contexts.

Experimental Evaluation

We evaluated NaRLE using a private dataset from an email-based digital assistant. The framework was tested on two distinct tasks:

  • Multi-class Intent Detection: Classifying emails into "modify," "cancel," or "other."
  • Multi-label Intent Detection: A complex task involving six possible actions.

To train the initial emotion detector, we utilized a semi-synthetic approach, injecting emotional feedback into email sentences to help the model distinguish between general sentiment and task-specific feedback.

Results and Discussion

Our empirical results demonstrate that NaRLE significantly enhances model performance. Fine-tuning pre-trained supervised models using RL led to improvements in intent classification accuracy of up to 43%.

Accuracy over time for multi-class and multi-label tasks, comparing RL variants against a supervised baseline.

Key findings include:

  • Robustness to Sparse and Noisy Feedback: NaRLE remains effective even when only 15% of users provide emotional cues and 33% of those cues are noisy (incorrectly detected).
  • Pre-training Advantage: While the agent can learn from scratch (random initialization), starting with a pre-trained supervised model results in faster convergence and higher peak accuracy.
  • Efficiency: DistilBERT was identified as the optimal backbone, providing high accuracy with the low latency necessary for real-time applications.

Conclusion

NaRLE provides a robust path for improving conversational assistants online while respecting user privacy and avoiding the costs of manual labeling. By treating human emotion as a proxy for task success, the system can adapt to real-world usage patterns. Future work will focus on more granular reward structures to handle "Neutral" feedback and improving emotion attribution to distinguish between assistant performance and external user factors.

Summary updated

Abstract

Current research in dialogue systems is focused on conversational assistants working on short conversations in either task-oriented or open domain settings. In this paper, we focus on improving task-based conversational assistants online, primarily those working on document-type conversations (e.g., emails) whose contents may or may not be completely related to the assistant's task. We propose "NARLE" a deep reinforcement learning (RL) framework for improving the natural language understanding (NLU) component of dialogue systems online without the need to collect human labels for customer data. The proposed solution associates user emotion with the assistant's action and uses that to improve NLU models using policy gradients. For two intent classification problems, we empirically show that using reinforcement learning to fine tune the pre-trained supervised learning models improves performance up to 43%. Furthermore, we demonstrate the robustness of the method to partial and noisy implicit feedback.
BibTeX
@misc{zhou2021narle,
  author = {Ruijie Zhou and Soham Deshmukh and Jeremiah Greer and Charles Lee},
  title = {{NaRLE: Natural Language Models using Reinforcement Learning with Emotion Feedback}},
  year = {2021},
  eprint = {2110.02148},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2110.02148},
}
Figure