Abstract
Prompt engineering—improving the command given to a large language model (LLM)—is becoming increasingly useful in order to maximize the performance of the model and therefore the quality of the output. However, in certain instances, the user is not able to enrich the prompt with additional and personalized details, such as the preferred tone and length of generated response. Therefore, it is useful to create models that learn these preferences and implement them directly in the prompt. Current state-of-the-art inductive logic programming (ILP) systems can play an important role in the development and advancement of digitalization strategies. For example, they can be used to learn personal preferences of users without sacrificing human interpretability of the learned outcomes. These systems have recently witnessed the development of data efficient, robust, and human interpretable algorithms and systems for learning predictive models from data and background knowledge. In this paper, one of these systems, ILASP (inductive learning of answer set programs), is used to develop a proof of concept of how personal preferences of groups of students participating in an interview exercise can be learned to tailor and improve the generated response of a LLM used in an educational context.
Introduction
While Large Language Models (LLMs) like ChatGPT have revolutionized natural language processing, they face significant hurdles in specialized domains such as education. Two primary issues are "hallucinations"—the generation of factually incorrect information—and a lack of personalization. In a traditional classroom, a teacher dynamically adjusts their pedagogical approach based on a student's confusion or communication style. Standard chatbots, by contrast, often provide generic, "one-size-fits-all" responses.
This research addresses these gaps within the context of a Good Manufacturing Practices (GMP) course at the Technical University of Denmark (DTU). Students engage in an "audit exercise" where they interview a fictional company. Previously conducted by human instructors, this role is now filled by "ChatGMP," a specialized bot. The goal is to enhance ChatGMP's ability to act as a mentor: detecting missed critical questions, identifying repetitive queries stemming from confusion, and adapting to specific user preferences.
Methodology and Data
The proposed framework bridges connectionist AI (LLMs) with symbolic AI (Logic Programming) to create a more robust educational tool.
The ChatGMP Pipeline
The system utilizes a FLAN-T5 base model for natural language processing. To mitigate hallucinations, it employs a Retrieval-Augmented Generation (RAG) approach, searching a curated corpus of documents to provide contextually accurate answers.
Logic-Based Prompt Enrichment
The core innovation lies in the integration of Inductive Learning of Answer Set Programs (ILASP). ILASP is used to extract human-readable logic rules from historical interaction data. For instance, if a student asks two highly similar questions consecutively, the system infers confusion. These logic rules are then translated into natural language instructions and appended to the LLM's prompt, dynamically altering its behavior to be more explanatory or concise.
Dataset
The study leveraged a real-world dataset consisting of 35 recorded audit exercises from the DTU GMP course (Spring 2022 and 2023). Each 45-minute session was processed using Python for audio extraction and diarization, followed by speech-to-text conversion. The resulting data was manually cleaned into a structured JSON format of question-answer pairs for analysis.
Key Insights and Logic Learning
A major advantage of using Inductive Logic Programming (ILP) is its inherent interpretability. Unlike "black-box" models, ILP produces rules that humans can read and verify.
- Similarity Thresholds: The system uses cosine similarity to evaluate student inquiries. A threshold (e.g., > 62) determines if a question is relevant to reference material. High similarity between consecutive questions (threshold > 60) triggers a more elaborate, simplified response mode to address potential student confusion.
- Weak Constraints for Preference Learning: By applying "weak constraints" in logic programming, the researchers identified that certain topics, such as "logistics" or "review," are stronger indicators of student performance. This allows the bot to nudge students toward critical areas they might be overlooking.
- Feedback Loops: The system evaluates the quality of student inquiries in real-time. If a student misses a crucial "reference question" necessary for a successful audit, the system can provide immediate feedback.
Results and Performance
The proof-of-concept successfully addressed three primary use cases:
- Performance Categorization: The system learned to accurately categorize student performance as "correct," "semi-correct," or "wrong" based on their coverage of key reference topics.
- Confusion Detection: By establishing mathematical thresholds for "looping" behavior, the bot effectively switched to a simplified explanatory mode when students appeared stuck.
- Preference Identification: ILASP identified correlations between student focus areas (among seven GMP topics) and their final grades. This enables the bot to provide proactive suggestions, such as recommending a focus on "maintenance" to improve a student's final report.
Conclusion and Future Directions
While the current implementation analyzes data a posteriori, it demonstrates the power of hybrid AI in education. Future work will focus on full, real-time integration into the ChatGMP pipeline. The researchers aim to unify "repetitive question" detection and "user preference" learning into a single engine, allowing the bot to adapt its tone, length, and complexity mid-conversation to meet the evolving needs of student groups.
Summary updated