AI-aided Decision-Making in Education - Uncertainty, Explainability, and Human Responsibility

Fabian Kieser, Paul Tschisgale, Peter WulffarXiv 2026 · DOI · arXiv

This paper, a preprint of a chapter for the upcoming book AI Roles and Responsibilities in Education, provides a conceptual framework for the ethical integration of AI into the classroom. Authors Fabian Kieser, Paul Tschisgale, and Peter Wulff argue that for AI to be a legitimate partner in education, we must move beyond "black box" implementations and ground the technology in three core principles: Uncertainty, Explainability, and Human Responsibility.

TL;DR

The paper argues that AI in education must transition from opaque "black boxes" to transparent systems by prioritizing uncertainty quantification, multi-layered explainability, and human-in-the-loop accountability. It provides a roadmap for stakeholders to ensure AI augments rather than replaces pedagogical expertise, especially in high-stakes scenarios like grading and admissions.

What Problem Does This Paper Address?

As AI systems—from Intelligent Tutoring Systems (ITS) to Large Language Models (LLMs)—permeate education, they are increasingly used for high-stakes decision-making. However, these systems are often opaque. When an AI grades an exam or flags a student as "at risk" of dropping out, the lack of transparency can lead to distrust, especially if the outcome seems biased or incorrect.

The authors highlight a looming regulatory challenge: the EU Artificial Intelligence Act (2024) classifies many educational AI applications as "high risk" because they can determine a person's educational and professional trajectory. The paper addresses the gap between the technical capabilities of AI and the ethical/legal necessity for these systems to be understandable and accountable to educators and students.

Methods

This is a conceptual synthesis and review paper. Rather than introducing a new algorithm, the authors:

  1. Categorize AI in Education: They adopt a taxonomy (learner-focused, teacher-focused, and institution-focused AI) to map the current landscape.
  2. Apply Statistical Critique: They use a hypothetical early warning system to demonstrate why standard metrics like "accuracy" are insufficient for educational ethics.
  3. Synthesize Explainability Frameworks: They apply Lipton’s (2018) taxonomy of explainability to the educational context, distinguishing between technical transparency and post-hoc interpretability.
  4. Illustrate Bias via Vector Space: They use word2vec embeddings to qualitatively demonstrate how training data can distort domain-specific semantic relationships (e.g., in physics).

Datasets

Not discussed in the paper. The authors use a hypothetical contingency table for illustrative purposes and reference the word2vec-google-news-30 model for a qualitative example of semantic bias.

Key Insights and Ideas

  • The "Accuracy" Trap: A system can have 93% accuracy but still be ethically "insufficient." In their hypothetical model, high accuracy masked a sensitivity of only 60%, meaning 40% of failing students were missed. They argue that precision and accuracy are averages, but educators need to know the uncertainty of a specific decision impacting a specific student.
  • Explainability is Multi-Faceted: The authors emphasize that "explainability" isn't one thing. It includes Transparency (how the engine is built) and Interpretability (how we make sense of the exhaust).
Figure 1. Taxonomy of the terms explainability, transparency, and interpretability developed by Lipton (2018).
  • LLMs Pose a New Challenge: Traditional explainability methods (like feature assignment) are often computationally prohibitive for LLMs with billions of parameters. This necessitates new "prompting strategies" to help these models explain their own reasoning.
  • Human Agency as a Right: The authors argue that students should have the right to access and even modify the "learner models" AI builds about them. This empowers students to challenge a model that might not accurately represent their growth or aspirations.

Key Results

The paper’s most striking quantitative illustration is the breakdown of a hypothetical early warning system. It demonstrates how a seemingly "good" model can be a "bad" tool:

Table 1. Contingency table illustrating the performance of the hypothetical AI-based early warn- ing system on the test data set.
  • Predictive Accuracy: 93% (Sounds great).
  • Sensitivity (Recall): 60% (Fails to identify 40% of students at risk).
  • Precision: 37.5% (Only ~1 in 3 students predicted to fail actually do).

This result underscores the authors' point: in high-stakes education, we must optimize for the right metric (like recall) and communicate the uncertainty of every individual prediction to the teacher.

Qualitatively, the authors show that word2vec models can be "distorted." For example, while physics + stars ≈ star, the model incorrectly associates general terms like "experiments" or "field" more closely with mechanics than electricity, reflecting biases in the news-based training data rather than physical reality.

Limitations and Future Work

  • The Performance-Explainability Trade-off: The authors acknowledge that complex models (deep neural networks) often perform better but are harder to explain than simpler, less accurate models (linear regression).
  • Regulatory Refinement: They suggest that the EU AI Act’s "high risk" categorization is currently too broad and needs a more refined approach to avoid stifling the development of low-stakes, beneficial tools.
  • Implementation Gap: While the paper urges developers to implement uncertainty quantification, it notes that this is still an active and difficult field of research, particularly for generative AI.

Summary updated

Abstract

As Artificial Intelligence (AI)-based systems have become increasingly integrated into various aspects of everyday life, including education, understanding how these systems make decisions is of paramount importance. AI holds significant potential to enhance educational practices through the facilitation of personalized and adaptive learning, automation of assessment procedures, and assistance with administrative tasks. However, the opacity of AI-based decision-making in education (especially with deep learning and generative AI models) can hamper decision-making, especially when outcomes appear unjustified or incorrect. In this chapter, we argue that three key principles (uncertainty, explainability, and human responsibility) are essential for the legitimate use of AI in educational settings. Acknowledging the inherent uncertainties in AI-based decisions, analogous to those in human decision-making, is fundamental. Explainability is critical for ensuring that the functioning and outcomes of AI applications are transparent and comprehensible to both educators and learners. Furthermore, human responsibility is crucial, particularly in human decision-making, which remains indispensable across all stages of AI development and deployment. By examining these three principles, this chapter provides a comprehensive guide for stakeholders seeking to integrate AI thoughtfully and effectively into the educational system.
BibTeX
@misc{kieser2026ai,
  author = {Fabian Kieser and Paul Tschisgale and Peter Wulff},
  title = {{AI-aided Decision-Making in Education - Uncertainty, Explainability, and Human Responsibility}},
  year = {2026},
  eprint = {2611.17858},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17858},
}
Figure