Introduction
In educational technology, "learner models" serve as computational agents that simulate student behavior. These models are essential for training educators and evaluating curricula without the logistical hurdles of recruiting human participants. However, developing effective learner agents presents a significant challenge. Traditional Reinforcement Learning (RL) agents excel at goal-oriented tasks but are linguistically "brittle," often failing when task descriptions are slightly rephrased. Conversely, Large Language Model (LLM) agents possess sophisticated conversational abilities but often lack "grounding," leading to hallucinations or failures to adhere to environment-specific constraints.
This research explores a hybrid approach to bridge this gap, aiming to create agents that generalize across linguistic variations while maintaining high task performance.
Methodology: Integrating RL and LLMs
The study evaluates three primary agent architectures:
- RL-based Agents: Using a Deep Reinforced Relevance Network (DRRN), these agents learn through trial and error, representing states and actions in natural language.
- LLM-based Agents: Utilizing GPT-4, these agents operate via prompting, including "reflective" versions that review past performance.
- Hybrid LLM-assisted RL Agents: The core innovation of this work, featuring two configurations:
- Suggestion-Assisted RL (SA-RL): The LLM filters the action space to suggest the top $k$ sensible actions, which the RL agent then evaluates to make a final choice.
- Decision-Assisted RL (DA-RL): The RL agent proposes the top $k$ actions, and the LLM selects the most contextually appropriate one.
The PharmaSimText Benchmark
To test these agents, the researchers developed PharmaSimText, a benchmark based on a virtual pharmacy simulator. Agents act as trainees diagnosing conditions like infant diarrhea or joint pain by interviewing virtual patients. To ensure robustness, the authors used GPT-4 to generate 10 diverse phrasings for every patient response, challenging the agents to generalize beyond simple keyword matching.
Results and Discussion
The evaluation focused on two metrics: Post-test Performance (diagnostic accuracy) and Trajectory Quality (the thoroughness of the diagnostic interview).
The results revealed a distinct trade-off. Pure RL agents acted as "speedrunners," achieving high accuracy (~0.9) but poor trajectory quality (~0.4) by skipping essential diagnostic questions. Pure LLM agents were "chatterboxes," maintaining high conversation quality (~0.8) but failing to reach the correct diagnosis consistently (~0.7).
The SA-RL hybrid model emerged as the superior architecture. By allowing the LLM to prune nonsensical actions while the RL agent focused on goal-oriented logic, SA-RL matched the RL agent's accuracy while significantly improving trajectory quality to ~0.8. Furthermore, SA-RL demonstrated strong generalization, maintaining performance even when patient responses were heavily rephrased—a scenario where traditional RL models typically degrade.
Conclusion
Integrating the linguistic intelligence of LLMs with the robust decision-making of RL creates learner agents that are both accurate and naturalistic. While this study focuses on wording generalization within pharmacy scenarios, future work will explore task generalization across different domains and the application of these agents in real-time teacher training environments.
Summary updated