On the Effectiveness of Prompt-Moderated LLMs for Math Tutoring at the Tertiary Level

Sebastian Steindl, Fabian Brunner, Nada Sissouno, Dominik Schwagerl, Florian Schöler-Niewiera, Ulrich SchäferEMNLP 2025

TL;DR

This paper investigates how Large Language Models (LLMs) perform as math tutors for university students, comparing unmoderated LLMs, prompt-moderated "tutor" LLMs, and no LLM assistance. Surprisingly, prompt-moderation seemed to hinder learning, while unmoderated LLMs led to better performance, even on unassisted exams, challenging concerns about shallow learning.

What Problem Does This Paper Address?

The rapid rise of LLMs like ChatGPT has opened doors for their use in education, but their effectiveness, especially in formal settings, remains a hot topic. While LLMs can provide quick answers, there's a concern they might lead to shallow learning, where students copy-paste without understanding. This paper specifically tackles how LLMs, when guided to act as tutors (prompt-moderated) versus when left to their own devices (unmoderated), impact tertiary-level students' math learning and performance, particularly in a controlled, exam-like environment. The authors aim to provide empirical evidence on the effectiveness of these different LLM tutoring approaches.

Methods

The researchers designed a three-phase study with 49 university math students.

  1. Review Phase: A lecturer reviewed relevant topics for all students.
  2. Exercise Phase: Students were divided into three groups:
  3. NON: Used only traditional coursework materials.
  4. GPT: Used an unmoderated GPT-4o-mini LLM.
  5. TUT: Used a prompt-moderated GPT-4o-mini LLM, instructed to act as a tutor and guide students without directly giving solutions. This phase involved solving math assignments, with LLM assistance available only for the GPT and TUT groups.

  6. Exam Phase: All groups took an exam without any LLM assistance.

The study analyzed student performance (scores and task completion), engagement with the LLM, and conversation strategies. They also introduced the GMADE (German Math Assistant Dataset for LLMs in Education), a collection of annotated student-LLM conversations.

Datasets

  • GMADE (German Math Assistant Dataset for LLMs in Education): This is a newly introduced dataset by the authors. It consists of 32 multi-turn German conversations between students and the GPT-4o-mini LLM during the exercise phase. The conversations are annotated with user message types (e.g., direct, abstract, follow-up), frustration signals, and mappings to specific assignments. This dataset aims to foster future research into LLM tutoring.

Key Insights and Ideas

  • Prompt-Moderation Backfires: The most striking finding is that the "tutor" LLM (prompt-moderated) actually led to worse performance compared to both the unmoderated LLM and even the group with no LLM assistance during the exercise phase.
  • Unmoderated LLMs Show Promise: The unmoderated LLM group consistently outperformed the other groups, not only during the assisted exercise but also during the unassisted exam. This suggests that LLMs, even without explicit tutoring prompts, can have a beneficial impact on learning.
  • Challenging Shallow Learning Concerns: The fact that LLM-assisted groups performed better on the unassisted exam than the control group directly contradicts fears that LLM use leads to shallow learning and an inability to perform without assistance.
  • Engagement Driven by Task, Not Topic: Student engagement with LLMs was more influenced by the question's type and length (e.g., easier to input, shorter questions) rather than the mathematical topic itself.
  • Frustration with Evasive Tutors: Students interacting with the prompt-moderated tutor LLM showed significantly higher levels of frustration, often trying to "trick" the LLM into giving direct answers.

Key Results

  • Exam Performance: The GPT (unmoderated LLM) group achieved the highest average scores on both the exercise and the exam. The TUT (prompt-moderated LLM) group performed similarly to the NON group during the exercise but surpassed them on the exam.
  • Task Completion: The GPT group had the highest task completion rate, while the TUT group had the lowest.
  • Frustration: The TUT group exhibited significantly more frustration signals (11.76% of messages) compared to the GPT group (2.58%).
  • Transfer Tasks: Contrary to expectations, LLM assistance did not lead to worse performance on transfer tasks when unassisted. The GPT group, in particular, showed strong performance on these tasks.

Limitations and Future Work

  • Prompt-Moderation Effectiveness: The study suggests that the specific prompt-moderation strategy used might be detrimental, and future work should focus on developing more beneficial prompt-moderation techniques.
  • Human vs. LLM Tutoring: The study did not compare LLM tutoring directly against human tutoring, only against no tutoring.
  • Dataset Size: The GMADE dataset, while valuable, is relatively small (32 conversations).
  • Time Constraints: The strict time limits for the exercise and exam phases might not perfectly reflect real-world study scenarios.
  • LLM Behavior: The Tutor LLM occasionally failed to adhere strictly to its prompt, sometimes providing direct solutions.

The authors suggest future work should explore how to make prompt-moderation beneficial, how to best integrate LLMs with traditional lectures, and the role of specific subject domains. They also highlight the need for further studies into LLM tutors, as their assistance can still have a positive effect on student performance.

Summary updated

BibTeX
@inproceedings{steindl2025effectiveness,
  author = {Sebastian Steindl and Fabian Brunner and Nada Sissouno and Dominik Schwagerl and Florian Schöler-Niewiera and Ulrich Schäfer},
  title = {{On the Effectiveness of Prompt-Moderated LLMs for Math Tutoring at the Tertiary Level}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure