Developing LLM-Powered Trustworthy Agents for Personalized Learning Support
Introduction
The integration of Large Language Models (LLMs) into educational settings offers transformative potential for personalized learning. However, standard AI models often struggle with factual accuracy and pedagogical appropriateness. MerryQuery is a novel LLM-powered educational agent designed to address these challenges. By combining Retrieval-Augmented Generation (RAG) with teacher-defined rule-based controls and Reinforcement Learning from Human Feedback (RLHF), MerryQuery provides a trustworthy tutoring environment that guides students toward solutions rather than simply providing them.
Problem Statement
Despite the promise of LLMs, three primary hurdles prevent their widespread adoption in classrooms:
- Trustworthiness: LLMs are prone to "hallucinations," where they fabricate facts or provide contextually irrelevant information.
- Pedagogical Misalignment: Standard AI often provides direct answers to homework questions, bypassing the critical thinking process and raising academic integrity concerns.
- Static Personalization: Many tools fail to adapt to individual student histories, learning paths, or evolving educational goals.
MerryQuery aims to bridge this gap by creating an agent that is context-aware, instructor-governed, and dynamically adaptive.
Methodology
The architecture of MerryQuery utilizes a multi-layered pipeline to ensure accuracy and educational value. The core components include:
- Retrieval-Augmented Generation (RAG): To mitigate hallucinations, the system grounds its responses in verified course materials (syllabi, assignments, and textbooks) stored in a dynamic vector database.
- Rule-Based Content Control: This "teacher-in-the-loop" feature allows instructors to set pre-retrieval and post-generation rules. For example, a rule might prevent the AI from providing full code solutions, ensuring it acts as a tutor rather than a ghostwriter.
- Guided Learning Dialogue: The agent is designed to provide "scaffolded hints" and problem-solving strategies, encouraging students to engage in critical thinking.
- Memory Mechanism & Dynamic Profiles: The system maintains a learning profile for each student, using historical interaction data to ensure coherent, long-term conversations tailored to individual progress.
Data and Implementation
MerryQuery was implemented and piloted in a real-world educational setting:
- Course-Specific Data: Materials from the CSC113 (Introduction to Computing - MATLAB) course at North Carolina State University, including syllabi and assignments, were used to populate the vector database.
- Student Interaction Data: Historical logs and student profiles were integrated to facilitate personalized support.
- Evaluation Framework: Future evaluations will utilize course forum data with verified answers as a "ground truth" for upcoming ablation studies.
Key Insights
Research during the development of MerryQuery revealed several critical insights:
- Prompt Engineering Effectiveness: Prompt engineering alone proved highly effective at enforcing teacher-defined rules, such as withholding full solutions. This reduces the computational overhead of separate post-generation verification agents.
- The "Tutor" Mindset: The innovation lies in making the AI more "disciplined." By focusing on scaffolding—a technique of providing temporary support—the agent mimics a human teaching assistant.
- Multimodal Context: The system is designed to handle diverse data types, recognizing that learning data includes complex PDFs, assignments, and historical performance metrics.
Results and Future Work
A functional prototype was successfully piloted at NC State, showing improved response reliability and better student engagement. The system effectively enforced policies against disclosing complete solutions, addressing a major concern for educators.
Future work will focus on:
- Refining Memory: Optimizing the memory mechanism for very long-term student interactions.
- Controlled Experiments: A scheduled study will compare MerryQuery against baseline LLMs using metrics like student engagement and perceived trustworthiness.
- Ablation Studies: Planned for early 2025, these studies will dissect the system to determine the relative contributions of RAG, RLHF, and rule-based controls to its overall success.
Summary updated