Towards Generalizable Agents in Text-Based Educational Environments: A Study of Integrating RL with LLMs

Bahar Radmehr, Adish Singla, Tanja Käser

Introduction

In educational technology, "learner models" serve as computational agents that simulate student behavior. These models are essential for training educators and evaluating curricula without the logistical hurdles of recruiting human participants. However, developing effective learner agents presents a significant challenge. Traditional Reinforcement Learning (RL) agents excel at goal-oriented tasks but are linguistically "brittle," often failing when task descriptions are slightly rephrased. Conversely, Large Language Model (LLM) agents possess sophisticated conversational abilities but often lack "grounding," leading to hallucinations or failures to adhere to environment-specific constraints.

This research explores a hybrid approach to bridge this gap, aiming to create agents that generalize across linguistic variations while maintaining high task performance.

Methodology: Integrating RL and LLMs

The study evaluates three primary agent architectures:

  1. RL-based Agents: Using a Deep Reinforced Relevance Network (DRRN), these agents learn through trial and error, representing states and actions in natural language.
  2. LLM-based Agents: Utilizing GPT-4, these agents operate via prompting, including "reflective" versions that review past performance.
  3. Hybrid LLM-assisted RL Agents: The core innovation of this work, featuring two configurations:
  4. Suggestion-Assisted RL (SA-RL): The LLM filters the action space to suggest the top $k$ sensible actions, which the RL agent then evaluates to make a final choice.
  5. Decision-Assisted RL (DA-RL): The RL agent proposes the top $k$ actions, and the LLM selects the most contextually appropriate one.
The architecture of the hybrid SA-RL and DA-RL agents, showing the interaction loops between the LLM and the RL-based agent.

The PharmaSimText Benchmark

To test these agents, the researchers developed PharmaSimText, a benchmark based on a virtual pharmacy simulator. Agents act as trainees diagnosing conditions like infant diarrhea or joint pain by interviewing virtual patients. To ensure robustness, the authors used GPT-4 to generate 10 diverse phrasings for every patient response, challenging the agents to generalize beyond simple keyword matching.

Results and Discussion

The evaluation focused on two metrics: Post-test Performance (diagnostic accuracy) and Trajectory Quality (the thoroughness of the diagnostic interview).

The results revealed a distinct trade-off. Pure RL agents acted as "speedrunners," achieving high accuracy (~0.9) but poor trajectory quality (~0.4) by skipping essential diagnostic questions. Pure LLM agents were "chatterboxes," maintaining high conversation quality (~0.8) but failing to reach the correct diagnosis consistently (~0.7).

Comparative performance of RL-based, LLM-based, and hybrid agents across different metric types.

The SA-RL hybrid model emerged as the superior architecture. By allowing the LLM to prune nonsensical actions while the RL agent focused on goal-oriented logic, SA-RL matched the RL agent's accuracy while significantly improving trajectory quality to ~0.8. Furthermore, SA-RL demonstrated strong generalization, maintaining performance even when patient responses were heavily rephrased—a scenario where traditional RL models typically degrade.

Conclusion

Integrating the linguistic intelligence of LLMs with the robust decision-making of RL creates learner agents that are both accurate and naturalistic. While this study focuses on wording generalization within pharmacy scenarios, future work will explore task generalization across different domains and the application of these agents in real-time teacher training environments.

Summary updated

Abstract

There has been a growing interest in developing learner models to enhance learning and teaching experiences in educational environments. However, existing works have primarily focused on structured environments relying on meticulously crafted representations of tasks, thereby limiting the agent's ability to generalize skills across tasks. In this paper, we aim to enhance the generalization capabilities of agents in open-ended text-based learning environments by integrating Reinforcement Learning (RL) with Large Language Models (LLMs). We investigate three types of agents: (i) RL-based agents that utilize natural language for state and action representations to find the best interaction strategy, (ii) LLM-based agents that leverage the model's general knowledge and reasoning through prompting, and (iii) hybrid LLM-assisted RL agents that combine these two strategies to improve agents' performance and generalization. To support the development and evaluation of these agents, we introduce PharmaSimText, a novel benchmark derived from the PharmaSim virtual pharmacy environment designed for practicing diagnostic conversations. Our results show that RL-based agents excel in task completion but lack in asking quality diagnostic questions. In contrast, LLM-based agents perform better in asking diagnostic questions but fall short of completing the task. Finally, hybrid LLM-assisted RL agents enable us to overcome these limitations, highlighting the potential of combining RL and LLMs to develop high-performing agents for open-ended learning environments.
BibTeX
@inproceedings{radmehr2024towards,
  author = {Bahar Radmehr and Adish Singla and Tanja Käser},
  title = {{Towards Generalizable Agents in Text-Based Educational Environments: A Study of Integrating RL with LLMs}},
  booktitle = {EDM},
  year = {2024},
  url = {https://openreview.net/forum?id=zwBq6Gwkai},
}
Figure