Learning to Diagnose Students’ Behavioral, Developmental, and Learning Disorders in a Simulation-Based Learning Environment for Pre-Service Teachers

Elisabeth Bauer, Michael Sailer, Jan Kiesewetter, Claudia Schulz, Iryna Gurevych, Martin R. Fischer, Frank FischerOpenalex Unknown 2022 · DOI · OpenAlex

TL;DR

The FAMULUS project introduces a simulation-based learning environment designed to help pre-service teachers diagnose behavioral and learning disorders in students. Its core innovation is an AI-driven feedback system that uses Natural Language Processing (NLP) to analyze a learner's written diagnostic reasoning and provide real-time, adaptive guidance on their decision-making process.

What Problem Does This Paper Address?

While teachers are often the "first responders" in identifying student disorders like ADHD or dyslexia, most report feeling ill-equipped for the task, often rating their diagnostic knowledge as "mediocre at best." Diagnostic competence isn't just about knowing a definition; it requires "epistemic activities"—generating hypotheses, gathering evidence, and drawing substantiated conclusions.

The challenge in higher education is that providing personalized, process-oriented feedback on these complex activities is incredibly time-consuming for instructors. This paper explores how to automate that feedback loop using simulation and AI, bridging the gap between theoretical knowledge and classroom practice.

Methods

The researchers developed a simulation environment called FAMULUS, hosted on the CASUS learning management system.

  1. Simulation Design: The environment uses a "serial cue format." Learners are presented with six cases of "virtual students" showing problematic behavior. Information is released sequentially (classroom observations, report cards, parent interviews), forcing learners to actively search for and evaluate evidence.
  2. The Task: Learners must generate initial hypotheses, gather data, and finally write a "substantiated argumentation text" explaining their diagnostic conclusion.
  3. The NLP Stack: To provide feedback on these free-text answers, the team trained an NLP algorithm using an Artificial Neural Network. Specifically, they combined Conditional Random Fields (CRFs)—which look at correlations between adjacent words—with Bidirectional Long Short-Term Memory (BiLSTM) networks, which capture long-term dependencies and context within the text.
  4. Feedback Logic: The system identifies the presence or absence of four key epistemic activities (hypothesizing, evidence generation, evidence evaluation, and concluding) to trigger predefined, adaptive feedback components.

Datasets

The algorithm was trained on a dataset of 944 argumentation texts collected from a previous study involving 118 pre-service teachers.

  • Manual Coding: Four human experts manually coded these texts to identify the four epistemic activities.
  • Training/Testing Split: A subset of 440 texts was used to fit the initial neural network model, while 110 additional texts were used to test the algorithm's performance.
  • Validation: 150 fourfold-coded texts were used to establish intercoder reliability.

Key Insights and Ideas

  • Process over Outcome: The authors argue that "process-related" feedback (explaining how to improve the diagnostic strategy) is significantly more effective for learning than "correct/incorrect" feedback.
  • Interdisciplinary Synergy: The project draws heavily from medical education. The concept of "virtual patients" was adapted into "virtual students," demonstrating that diagnostic reasoning is a cross-domain skill that can be taught using similar architectural frameworks.
  • The "Serial Cue" Advantage: By presenting information in units rather than all at once, the simulation mimics the real-world "uncertainty reduction" process that professionals face.
  • NLP as a Bridge: The paper highlights that while NLP was previously limited to analyzing writing strategy (lexical/rhetorical features), modern neural networks allow for the analysis of content and logic in professional reasoning.

Key Results

  • High Accuracy: The FAMULUS algorithm achieved more than 70% of human intercoder reliability in identifying epistemic activities.
  • State-of-the-Art Performance: The system achieved nearly twice the performance reported by previous studies attempting to automate the identification of these specific diagnostic activities.
  • Validation: The cases were validated by experts in school psychology and psychotherapy to ensure "symptomatic authenticity," ensuring the simulation remains a high-fidelity representation of real-world challenges.

Limitations and Future Work

  • Data Scarcity: The initial model was trained on a relatively small subset (440 texts). The authors plan to extend this to the full 944-text dataset to improve accuracy.
  • Instructional Clarity: The authors noted that the quality of the NLP output depends on the quality of the input text. They suggest that clarifying task instructions for students could lead to more consistent argumentation, which in turn makes the AI more accurate.
  • Experimental Validation: A laboratory study is planned to compare this adaptive AI feedback against static "expert solution" feedback to quantify the actual learning gains.
  • Broadening Scope: Future iterations will attempt to automate the coding of "case-specific concepts" (e.g., identifying specific symptoms of ADHD) in addition to the general diagnostic strategy.

Summary updated

Abstract

Abstract Reasoning about students who might have behavioral, developmental, or learning disorders is a relevant aspect of teachers’ everyday practice (Reinke et al., Sch Psychol Q 26(1):1, 2011). Therefore, this content area should be part of teacher education. Accordingly, we developed a simulation-based learning environment in which pre-service teachers gather information about six individual students. Learners examine reports about students’ behavior, e.g., in the classroom or at home, and analyze the students’ performance and records of their work. The pre-service teachers’ task is to integrate the given information and draw a diagnostic conclusion for each student. Several design aspects were investigated using the simulation. The most challenging design aspect was to automatically generate adaptive feedback on epistemic diagnostic activities and diagnostic outcomes (Schulz et al., e-teaching. org Themenspecial, Was macht Lernen mit digitalen Medien erfolgreich, 2019). The studies we conduct are replicated in a parallel project in medical education.
BibTeX
@inproceedings{bauer2022learning,
  author = {Elisabeth Bauer and Michael Sailer and Jan Kiesewetter and Claudia Schulz and Iryna Gurevych and Martin R. Fischer and Frank Fischer},
  title = {{Learning to Diagnose Students’ Behavioral, Developmental, and Learning Disorders in a Simulation-Based Learning Environment for Pre-Service Teachers}},
  booktitle = {Openalex Unknown},
  year = {2022},
  doi = {10.1007/978-3-030-89147-3\_8},
}

Loading related content...

Figure