TL;DR
This paper introduces KELE, a novel multi-agent framework that uses Large Language Models (LLMs) to automate structured Socratic teaching. By employing a "consultant-teacher" architecture and a rule-based system, KELE aims to provide coherent, progressive, and effective Socratic dialogues, outperforming even GPT-40 on Socratic teaching capabilities.
What Problem Does This Paper Address?
Traditional Socratic teaching, while effective for fostering deep thinking, is highly demanding on teacher expertise and real-time feedback, making it difficult to scale. Existing attempts to automate this with LLMs often focus on superficial dialogue styles and lack structured teaching processes, leading to disorganized interactions and incomplete learning consolidation. This paper addresses the need for a structured, controllable, and scalable Socratic teaching system powered by LLMs.
Methods
The core of the KELE framework is a two-agent system:
- Consultant Agent (Ac): Acts as the planner, analyzing student states and dialogue history to recommend teaching actions based on a structured rule system (SocRule).
- Teacher Agent (At): Executes the teaching plan by generating Socratic questions and feedback, guided by the consultant.
SocRule is a structured rule system that divides Socratic teaching into five progressive stages (student questioning, concept probing, inductive reasoning, rule construction, teacher summary) and defines 34 specific teaching strategies. This system ensures a coherent and hierarchical teaching process.
The framework also includes a "consultant-teacher" multi-agent collaborative teaching mechanism to manage the interaction between the two LLM agents, ensuring effective implementation of SocRule.
Datasets
- Guiding Problem-Solving Dataset: A foundational dataset of 6,803 elementary science questions with structured solutions, hints, knowledge points, and problem-solving thoughts, used to inform the construction of the main dataset.
- SocratDataset: A newly constructed, structured Socratic teaching dialogue dataset containing over 42,000 dialogue turns across 34 teaching strategies, designed to train LLMs for Socratic teaching. This dataset is built upon the guiding problem-solving dataset and SocRule.
Key Insights and Ideas
- Structured Socratic Teaching is Crucial: Simply mimicking Socratic dialogue isn't enough; a structured, rule-based approach is essential for effective and coherent teaching.
- Multi-Agent Collaboration Enhances Control: The "consultant-teacher" architecture effectively separates planning and execution, leading to more controlled and less random teaching dialogues compared to single-agent approaches.
- Specialized LLMs Outperform Generalists: Training a dedicated LLM (SocratTeachLLM) on a Socratic-specific dataset yields superior performance in Socratic teaching tasks, even against much larger general-purpose models.
- Comprehensive Evaluation is Necessary: Developing a multi-dimensional evaluation system is key to accurately assessing the nuanced capabilities of LLMs in Socratic teaching.
Key Results
- SocratTeachLLM Performance: The trained SocratTeachLLM significantly outperforms its base model (GLM4-9B) and even GPT-40 across all Socratic teaching capabilities.
- Quantitative Improvements: SocratTeachLLM shows substantial improvements in metrics like Problem Relevance Rate (PRR), No Direct Answer Rate (NDAR), Summary Pass Rate (SPR), and Instruction Adherence Rate (IAR) compared to GPT-40. For example, it achieved a 22.67% improvement in PRR over GPT-40.
- Evaluation System Effectiveness: The proposed 9-dimensional evaluation system demonstrates generalizability and systematicity, proving effective in assessing LLM performance in Socratic teaching.
Limitations and Future Work
- Dataset Domain Limitation: The current SocratDataset is based on scientific disciplines, which may limit the applicability of the trained SocratTeachLLM to other subjects. Future work will involve expanding datasets across more disciplines.
- Rule Adherence: Like most LLMs, SocratTeachLLM still struggles with consistently adhering to rules in complex Socratic teaching tasks, although the "consultant-teacher" mechanism helps mitigate this. Further enhancements are needed for improved rule compliance.
- Future Directions: The authors plan to continue optimizing the KELE framework and explore its applications in diverse educational scenarios and disciplines.
Summary updated