LLM-based Collaborative Agents with Pedagogy-guided Interaction Modeling for Timely Instructive Feedback Generation in Task-oriented Group Discussions

Qihao Yang, Yu Yang, Sixu An, Tianyong Hao, Guandong XuIJCAI 2025 · DOI · OpenReview

TL;DR

This paper introduces an LLM-based agent designed to act as an AI tutor in task-oriented group discussions (ToGD). It leverages pedagogical theories and a novel multi-agent simulation to generate timely, instructive feedback that guides students without disrupting their collaborative learning process, achieving state-of-the-art performance.

What Problem Does This Paper Address?

Traditional AI tutors excel at supporting individual learning but struggle with collaborative learning (CL) environments, particularly task-oriented group discussions (ToGD). Existing AI tutors often fail to model the dynamic interactions between students and tutors, leading to ineffective scaffolding, over-instruction, or a lack of timely, relevant feedback. This can hinder students' ability to develop critical thinking and teamwork skills, ultimately impacting learning outcomes. The core challenge is to create an AI agent that can understand the nuances of group discussions, identify learning issues, and provide pedagogical guidance that enhances, rather than disrupts, collaboration.

Methods

The authors propose a novel LLM-based collaborative agent with a multi-pronged approach:

  • CoI-based Prompting Strategy: They ground the agent's understanding of discussions in the Community of Inquiry (CoI) theory. This involves defining specific discussion stages (problem defining, exploration, integration, resolution, reflection), common learning issues within each stage, intervention conditions, and corresponding feedback rules. This structured approach allows the LLM to "sense" the discussion's status and potential problems.
  • Multi-Agent Interaction Framework: To learn how AI tutor feedback impacts student collaboration, they simulate ToGD using a multi-agent system. This framework allows the LLM tutor to interact with multiple student agents, generating synthetic dialogue data that captures collaborative behaviors and the effects of interventions.
  • Direct Preference Optimization (DPO): Instead of standard supervised fine-tuning (SFT), which maps an input to a single correct output, the authors use DPO. This method trains the LLM to prefer one response over another, allowing it to learn more nuanced decision-making and pedagogical adaptability, mimicking how human teachers might choose between different feedback strategies.

Datasets

  • Delidata: A dataset of 500 tutorless group discussions (14,000 utterances) used for initial post-annotation with the CoI framework.
  • CLTeach: A newly introduced synthetic dataset comprising 27,000 manually verified multi-party dialogues. This dataset was generated using the multi-agent simulation framework and includes fine-grained annotations for instructive feedback and explanations. It serves as the primary dataset for fine-tuning the final agent.

Key Insights and Ideas

  • Pedagogical Theory is Crucial: Simply fine-tuning LLMs on discussion data isn't enough. Integrating established pedagogical frameworks like CoI provides the necessary structure for the AI to understand and act effectively in a learning context.
  • Simulating Collaboration is Key: Learning the impact of AI feedback requires observing how students react to it. The multi-agent simulation is vital for generating data that captures these dynamic interactions, which is difficult to obtain from real-world settings.
  • Preference-Based Learning for Nuance: DPO proves more effective than SFT for this task, enabling the agent to learn more flexible and human-like decision-making in providing feedback, rather than being restricted to a single "correct" response.
  • Timeliness and Non-Disruption are Paramount: The success of an AI tutor in CL hinges on providing feedback at the right moment without interrupting the flow of student collaboration. The proposed agent aims to strike this delicate balance.

Key Results

The proposed LLM-based collaborative agent achieved state-of-the-art (SOTA) performance across various metrics on both the Delidata and CLTeach datasets.

  • Improved Intervention, Stage, and Issue Detection: The agent showed significant improvements over baseline LLMs like GPT-40, with gains of up to 38.17% in issue detection and 28.69% in stage identification on Delidata.
  • Superior Feedback Quality: The agent demonstrated a 24.44% improvement in BERTScore for feedback quality compared to GPT-40 on Delidata.
  • Effective CL Facilitation: In simulated discussions, the agent promoted smoother stage transitions (e.g., 10.91% for Stage 1 to Stage 2) compared to GPT-40, suggesting it guides students through the learning process more effectively.
  • Reduced Over-Instruction: The agent's guidance was less frequent than GPT-40's, with more utterances between interventions, indicating a better balance and less risk of over-instruction, thereby promoting student autonomy.
  • Enhanced Shared Metacognition: Visualizations showed that students guided by the agent exhibited positive cognitive transformations, moving from individual learning awareness to better group learning engagement and contribution.

Limitations and Future Work

  • Real-World Validation Needed: The authors acknowledge that the agent's effectiveness has not yet been validated in authentic classroom settings.
  • Learning Outcome Measurement: The current study focuses on feedback generation and interaction quality, not directly on improving students' academic performance or learning outcomes.

Future work will involve conducting in-class trials to validate the agent's effectiveness in real-world scenarios and collecting student data to measure actual learning outcome improvements.

Summary updated

Abstract

Large language models (LLMs) fundamentally reshape learning and teaching models, shifting tutoring systems from supporting individual learning to facilitating collaborative learning (CL) like task-oriented group discussions. However, existing AI tutors struggle to guide CL, as they seldom model the interactions between AI tutors and students. Therefore, they cannot scaffold students to complete tasks collaboratively, which impairs learning outcomes and pedagogy adaptability. Additionally, existing AI tutors fail to make use of CL theories to generate instructive feedback, which leads to undesirable interactions such as over-instruction and limits students' autonomy. In this paper, we propose an LLM-based collaborative agent that innovatively leverages pedagogical strategies to sense discussion stages, detect learning issues, identify the timing of intervention, and generate instructive feedback. To develop the agent, we first design a prompting strategy based on a CL theory, that is, the Community of Inquiry, to cultivate the agent to understand the discussion status. Second, a multi-agent interaction framework is proposed to simulate the collaborative learning behavior between AI tutors and students. Meanwhile, a synthetic task-oriented group discussion dataset, namely CLTeach, is generated, which consists of 27k manually-verified multi-party dialogues with fine-grained annotations of instructive feedback and explanations. Lastly, we use CLTeach to fine-tune the LLM agent, ultimately enabling it to generate instructive feedback at the right time to support students in CL. Extensive experiments demonstrate that our agent achieves state-of-the-art performance in feedback generation and has the potential to mimic human teachers effectively.
BibTeX
@inproceedings{yang2025llm,
  author = {Qihao Yang and Yu Yang and Sixu An and Tianyong Hao and Guandong Xu},
  title = {{LLM-based Collaborative Agents with Pedagogy-guided Interaction Modeling for Timely Instructive Feedback Generation in Task-oriented Group Discussions}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/1108},
}
Figure