To Align Long, Plan Short: Simulating Diverse Profile-Aligned Students by Realizing Short-Term Plans in Long-Horizon Learning Environments

Bahar Radmehr, Jiaxu Zhao, Fatma Betül Güreş, Adish Singla, Tanja KäserarXiv 2026 · arXiv

TL;DR

Simulating realistic student behavior in open-ended learning environments is notoriously difficult because LLMs tend to "drift" from their assigned personas over long interactions. This paper introduces ALPS (Align Long-term by Planning Short-term), a hierarchical architecture that uses a high-level planner to set segment-level goals and a low-level generator to execute actions. By planning at the segment level rather than turn-by-turn, ALPS maintains consistent behavioral profiles over hundreds of actions, even when training data is extremely scarce.

What Problem Does This Paper Address?

To build better AI tutors and learning platforms, researchers need to test them against diverse student behaviors—students who make specific mistakes, explore aimlessly, or follow systematic inquiry strategies. However, collecting real student data is expensive, slow, and fraught with privacy concerns.

While Large Language Models (LLMs) are great at simulating short dialogues, they struggle with long-horizon trajectories in Open-Ended Learning Environments (OELEs). In these settings, a student might take hundreds of actions. If you simply prompt an LLM turn-by-turn (e.g., "You are a struggling student, what is your next move?"), the model eventually suffers from profile drift: it loses track of its assigned persona and starts acting like a generic, helpful assistant or an idealized learner.

The authors identify three core challenges:

  1. Consistency: Maintaining a profile over hundreds of steps.
  2. Observability: Generic traits (like "introverted") are hard to verify in action logs; we need environment-specific behavioral profiles.
  3. Data Scarcity: New learning environments often have very few real student traces to learn from.

Methods

The researchers propose ALPS, a hierarchical planner-generator framework designed to anchor behavior to specific "profiles" derived from real-world data.

Figure 1: Overview of the proposed simulation framework. (1) Real student trajectories are clustered into environment-specific behavior profiles. (2) Conditioned on a target profile, ALPS generates long trajectories through a hierarchical planner-generator architecture. (3) When real trajectories are available, ALPS is adapted under both data-sufficient and data-scarce settings using ICL, SFT, and RAFT.

The process works in three main stages:

  1. Defining Profiles: They cluster real student trajectories based on behavioral features (like how often they test a hypothesis) to create distinct "behavioral signatures."
  2. The ALPS Architecture:
    • Segment Planner: Instead of picking one action, it plans a "segment" of behavior (e.g., "perform 5 low-quality explorations and then try a challenge question").
  3. Action Generator: This component takes the plan and realizes it through individual, valid environment actions.
  4. Adaptation: The models are refined using three techniques:
  5. In-Context Learning (ICL): Providing examples in the prompt.
  6. Supervised Fine-Tuning (SFT): Training on real or teacher-generated trajectories.
  7. Reward-Ranked Fine-Tuning (RAFT): Generating many trajectories and keeping only those that pass a "judge" (either a data-driven classifier or a teacher LLM).

Datasets

The authors evaluated ALPS on two distinct OELEs:

  • TugLet: An inquiry-based physics game where students experiment with tug-of-war teams to learn about force and net motion. Trajectories can reach up to 185 actions.
  • PharmaSim: A pharmacy simulation where students practice diagnostic conversations with patients and consult medical resources.

They tested two supervision settings: Data-Sufficient (using 140 real trajectories) and Data-Scarce (using only 3 real trajectories per profile, supplemented by a "teacher" LLM).

Key Insights and Ideas

  • Hierarchy Defeats Drift: The most important "aha" moment is that planning at the segment level (every 5-10 actions) acts as a "behavioral anchor." It prevents the model from re-deciding its persona at every single step, which is where the drift usually creeps in.
  • SFT is the Heavy Lifter: While RAFT (reinforcement learning style filtering) is popular, the authors found that Supervised Fine-Tuning (SFT) provided the most significant and reliable gains in both density and coverage.
  • Teacher LLMs as "Bridge" Supervisors: In data-scarce settings, a large "teacher" model (like GPT-4) can evaluate candidate trajectories against a checklist. Fine-tuning a smaller student model (like Qwen-8B) on these teacher-approved examples allows the smaller model to perform nearly as well as the teacher.
  • Co-adaptation: The planner and generator adapters work best when trained together. If you use a fine-tuned generator with a basic prompted planner, performance often collapses because the generator expects the specific types of plans it saw during training.

Key Results

ALPS consistently outperformed standard turn-level LLM generation across almost all metrics.

  • Profile Consistency: In the PharmaSim environment (Data-Sufficient), ALPS achieved a 95.6% judge pass rate, compared to only 39.6% for the turn-level baseline.
  • Behavioral Density: ALPS trajectories were much closer to real student data in the feature space. In TugLet, ALPS improved density from 48.7% to 83.9% over the baseline.
  • Data Scarcity Resilience: Even with only 3 real examples per profile, ALPS (using teacher-generated demonstrations) recovered most of the performance of the data-sufficient models.
Experimental results for the Data-Sufficient setting (Exp1).

Limitations and Future Work

  • Manual Design: Currently, the "segmentation rules" (deciding where one segment ends and another begins) and the "plan fields" are hand-designed for each environment. Future work could involve LLMs automatically discovering these structures.
  • Coverage Gap: While ALPS is very good at being consistent (density), it still struggles to capture the full breadth of human variation (coverage) in data-scarce settings. It tends to produce the "most typical" behavior for a profile rather than the full range of weird things humans do.
  • Generalization: The current study focuses on two specific environments. Testing how well these hierarchical planners transfer across different types of learning tasks remains an open question.

Summary updated

Abstract

Simulated students offer a scalable way to test learning platforms and AI tutors before deployment with real students and to augment scarce data for studying learning at scale. Existing LLM-based simulators primarily target short, language-centered interactions. In open-ended learning environments, however, student behavior emerges from patterns across long-horizon interaction trajectories, where turn-level prompting can drift from an assigned behavior profile. We introduce ALPS (Align Long-term by Planning Short-term), a hierarchical simulated student conditioned on observable, environment-specific behavior profiles. Rather than selecting every action independently, ALPS alternates between planning profile-aligned task segments and generating the corresponding actions. We evaluate ALPS in both data-sufficient and data-scarce settings, where supervision comes either from real student trajectories or from a teacher LLM. Across both settings, we compare in-context learning (ICL), supervised fine-tuning (SFT), reward-ranked fine-tuning (RAFT), and their sequential combination (SFT followed by RAFT). Experiments in an inquiry-based physics game and a pharmacy simulation evaluate behavioral density and coverage against held-out real data, and profile consistency using a human-aligned LLM judge. Across both settings, ALPS consistently outperforms turn-level generation, achieving comparable density and judge pass rates under scarce supervision, although coverage remains lower. These results show that hierarchical planning enables faithful, profile-aligned simulation over long interaction horizons, even with limited real student data.
BibTeX
@misc{radmehr2026align,
  author = {Bahar Radmehr and Jiaxu Zhao and Fatma Betül Güreş and Adish Singla and Tanja Käser},
  title = {{To Align Long, Plan Short: Simulating Diverse Profile-Aligned Students by Realizing Short-Term Plans in Long-Horizon Learning Environments}},
  year = {2026},
  eprint = {2611.02002},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.02002},
}
Figure