Abstract
Our proposed system FAMULUS helps students learn to diagnose based on automatic feedback in virtual patient simulations, and it supports instructors in labeling training data. Diagnosing is an exceptionally difficult skill to obtain but vital for many different professions (e.g., medical doctors, teachers). Previous case simulation systems are limited to multiple-choice questions and thus cannot give constructive individualized feedback on a student's diagnostic reasoning process. Given initially only limited data, we leverage a (replaceable) NLP model to both support experts in their further data annotation with automatic suggestions, and we provide automatic feedback for students. We argue that because the central model consistently improves, our interactive approach encourages both students and instructors to recurrently use the tool, and thus accelerate the speed of data creation and annotation. We show results from two user studies on diagnostic reasoning in medicine and teacher education and outline how our system can be extended to further use cases.
Introduction
Diagnostic reasoning—the ability to gather evidence and form a conclusion—is a critical skill for doctors, teachers, and engineers alike. However, it is notoriously difficult to teach at scale. Traditional online simulations often rely on multiple-choice questions, which only evaluate the final answer (the "what") rather than the underlying thought process (the "how"). While open-ended text explanations provide better insight into a student's reasoning, they are incredibly labor-intensive for instructors to grade. FAMULUS addresses this "cold-start" problem by integrating a neural sequence labeling pipeline with a "human-in-the-loop" annotation workflow.
System Architecture and Methods
The FAMULUS system consists of three intercommunicating components designed to bridge the gap between student learning and data annotation:
- CASUS: A student-facing simulation environment where users interact with "virtual patients" and write out their diagnostic process.
- INCEpTION: An interactive annotation platform where instructors review and label student texts.
- NeuralWeb: The "brain" of the system, a Python-based web service that handles model training and inference.
The system treats diagnostic reasoning as a sequence labeling task. It identifies diagnostic entities (e.g., "liver values") and epistemic activities, such as Hypothesis Generation (HG), Evidence Generation (EG), Evidence Evaluation (EE), and Drawing Conclusions (DC). The underlying model is a BiLSTM-CRF network, which is model-agnostic and can be updated as NLP technology evolves.
Evaluation and Key Results
The researchers evaluated FAMULUS using two distinct domains: Medicine (1,107 texts) and Teacher Education (944 texts). The BiLSTM achieved Macro-F1 scores ranging from 57.16 to 80.20 across different reasoning activities. While a gap remains compared to human experts, the model is accurate enough to provide meaningful guidance.
A critical requirement for classroom use is real-time performance. As shown in Figure 6, inference and feedback generation averaged 3.15 seconds per text, with a maximum delay of under 9 seconds.
Furthermore, the "warm-run" workflow significantly improved instructor efficiency. Using model-generated suggestions provided a speedup of 34 to 42 seconds per text for instructors, who accepted roughly 54-56% of the model's suggestions.
Discussion and Insights
FAMULUS demonstrates that process-oriented feedback is viable at scale. Because the model labels specific reasoning activities, it can tell a student exactly where their logic failed—for example, if they jumped to a conclusion without evaluating a specific lab result.
However, limitations remain. The "Feedback DB" containing expert-written snippets must still be manually curated for every new case study. Additionally, moving to new fields like law or engineering requires defining new annotation schemas. Despite these challenges, the system's modular architecture and interactive loop encourage recurrent use, accelerating the speed of data creation and improving feedback quality over time.
Summary updated