FAMULUS: Interactive Annotation and Feedback Generation for Teaching Diagnostic Reasoning

Jonas Pfeiffer, Christian M. Meyer, Claudia Schulz, Jan Kiesewetter, Jan M. Zottmann, Michael Sailer, Elisabeth Bauer, Frank Fischer, Martin R. Fischer, Iryna Gurevych

Introduction

Diagnostic reasoning—the process of gathering evidence to reach a conclusion—is a critical skill across diverse domains such as medicine, education, and engineering. However, teaching this skill at scale presents significant challenges. While virtual patient simulations are common, they often rely on multiple-choice questions that evaluate only the final diagnosis rather than the underlying reasoning process. Open-ended textual explanations offer deeper insight but create a grading bottleneck for instructors.

The FAMULUS system addresses this by using Natural Language Processing (NLP) to analyze students' diagnostic reasoning in real-time. It employs a "human-in-the-loop" architecture that provides individualized feedback to students while assisting instructors in creating the labeled datasets necessary for training automated models.

System Architecture

FAMULUS integrates three core components into a unified workflow designed to bridge the gap between manual annotation and automated feedback.

  1. NeuralWeb: A Python-based web service serving as the NLP engine. It utilizes BiLSTM architectures (via Keras and Flair/PyTorch) to perform sequence labeling on student text.
  2. INCEpTION: An interactive annotation platform where experts review and validate model-generated suggestions, accelerating the data labeling process.
  3. CASUS: A front-end simulation tool where students interact with virtual cases and submit their diagnostic explanations.
The FAMULUS system architecture illustrates the data flow between students (CASUS), instructors (INCEpTION), and the NLP core (NeuralWeb).

This architecture supports a "warm-run" loop: as instructors validate model suggestions in INCEpTION, the model is retrained, leading to higher accuracy and more reliable automated feedback for students in CASUS.

Identifying Epistemic Activities

To provide process-oriented feedback, FAMULUS identifies four key "epistemic activities" within student texts:

  • Hypothesis Generation (HG): Proposing potential diagnoses.
  • Evidence Generation (EG): Identifying relevant observations or data points.
  • Evidence Evaluation (EE): Assessing the relationship between evidence and hypotheses.
  • Drawing Conclusions (DC): Synthesizing evidence to reach a final decision.

By segmenting text into these categories, the system can pinpoint specific gaps in a student's logic, such as reaching a conclusion without sufficient evidence evaluation.

Evaluation and Key Results

The system was tested using two datasets: Medicine (Med), containing 1,107 texts from medical students, and Teacher Education (TEd), with 944 texts from education students.

Annotation Efficiency and Model Performance

The "human-in-the-loop" approach significantly improved instructor productivity. By providing automated suggestions, the system saved instructors between 34 and 42 seconds per text. Instructors accepted approximately 54-56% of these suggestions, demonstrating that even early-stage models provide valuable assistance. The BiLSTM models achieved Macro-F1 scores between 57 and 80, depending on the specific activity and domain.

Real-Time Performance

For classroom utility, feedback must be near-instant. The authors optimized the NLP pipeline to ensure low latency. On standard hardware, the average inference time for a student text was approximately 3.15 seconds, well within the requirements for interactive learning environments.

A scatter plot showing the relationship between text length and processing time, highlighting the system's ability to provide feedback in real-time.

Conclusion and Future Work

FAMULUS demonstrates a viable path for scaling process-oriented feedback in professional education. By combining interactive annotation with state-of-the-art NLP, it reduces the burden on instructors while providing students with nuanced insights into their diagnostic reasoning. Future work will focus on closing the performance gap between models and human annotators, particularly in complex domains like teacher education, and expanding the system to other technical fields.

Summary updated

BibTeX
@inproceedings{pfeiffer2019famulus,
  author = {Jonas Pfeiffer and Christian M. Meyer and Claudia Schulz and Jan Kiesewetter and Jan M. Zottmann and Michael Sailer and Elisabeth Bauer and Frank Fischer and Martin R. Fischer and Iryna Gurevych},
  title = {{FAMULUS: Interactive Annotation and Feedback Generation for Teaching Diagnostic Reasoning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=JUa80daXx_u},
}
Figure