Protecting and Promoting Human Agency in Education in the Age of Artificial Intelligence

Olga Viberg, Kasia Müldner, René F. Kizilcec, Simon Buckingham Shum, Dorottya Demszky, Dragan Gašević, Ian McInnis, Ioana Jivet, Jelena Jovanovic, Jennifer Meyer, Kou Murayama, Zach A Pardos, Chris Piech, Nikol Rummel, Naomi WinstonearXiv 2026 · DOI · arXiv

Abstract

Human agency is crucial in education and increasingly challenged by the use of generative AI. This meeting report synthesizes interdisciplinary insights and conceptualizes four aspects that delineate human agency: human oversight, AI-human complementarity, AI competencies, and relational emergence. We explore practical dilemmas for protecting and promoting agency, focusing on normative constraints, transparency, and cognitive offloading, and highlight key tensions and implications to inform ethical and effective AI integration in education.

Introduction

As generative artificial intelligence (AI) becomes deeply embedded in educational environments, a fundamental question arises: how do we protect human agency—the capacity to act intentionally and exert control over one's environment? The rapid adoption of large language models (LLMs) risks "outsourcing" the very cognitive and metacognitive skills that education is designed to cultivate. If AI handles the drafting, feedback, and assessment, the agency of both students and teachers may be severely diminished.

This paper synthesizes insights from a 2025 interdisciplinary workshop at Marbach Castle, Germany, involving experts from computer science, learning sciences, psychology, and philosophy. It introduces a framework to navigate the "System 0" era—a state where AI acts as an external cognitive layer that processes information before it reaches human consciousness, potentially framing choices and manipulating cognition before critical thinking even begins.

A Four-Dimensional Framework for Agency

To move beyond viewing AI as a mere tool and toward a model of AI as a partner, the authors propose four distinct dimensions of agency:

1. Human Oversight of AI

Agency is maintained only when humans retain the moral and operational authority to override AI systems. This requires "meaningful human control," ensuring that AI actions remain subordinate to human intentions rather than dictating the educational path.

2. Human-AI Complementarity

Rather than a simple division of labor where AI handles "boring" tasks, the report advocates for "Hybrid Adaptivity." This design philosophy ensures AI supports high-order thinking and ethical decision-making, rather than just automating low-level cognitive processes.

3. AI Competencies

Agency is not solely a technical property; it is a human capacity. This involves "AI literacy"—the ability to know when to trust, challenge, or ignore an output—and the intrinsic motivation to remain a reflective agent rather than a passive recipient of AI-generated content.

4. Relational Emergence

Agency does not exist in a vacuum; it emerges from the interaction between individuals, tools, and institutions. This "systemic agency" perspective considers how an AI assistant might alter the dynamics between students, parents, and teachers, shifting the locus of control within the entire educational ecosystem.

The integration of AI into education presents three central dilemmas that researchers and developers must address:

  • Normative Constraints: There is a tension between "hard-coding" educational values into AI and the risk of standardizing thought. While constraints can protect agency by guiding students toward pedagogical goals, they may also create filter bubbles that limit intellectual diversity.
  • The Explainability Trade-off: While transparency is a goal, complex models are often inherently opaque. The report suggests a dual approach: continuing the pursuit of explainable AI while simultaneously training humans to be "resilient in ambiguity," learning to work productively with "black box" systems.
  • The Zero-Sum Fallacy: The authors challenge the assumption that AI efficiency necessarily reduces human reflection. If AI manages routine administrative or feedback tasks, it may actually free up teacher agency, allowing for deeper, one-on-one mentoring and human connection.

Challenges and Future Directions

The path toward agency-promoting AI faces significant hurdles. Institutional pressures often prioritize cost-cutting and full automation over designs that support human agency. Furthermore, effective human oversight often requires extensive data collection, creating a direct tension with student privacy.

Finally, the report highlights a critical lack of longitudinal data. Most current insights are based on short-term interactions. Future research must investigate how "System 0" affects cognitive development over years, ensuring that we do not inadvertently trade long-term intellectual growth for short-term efficiency.

Conclusion

Safeguarding human agency in the age of AI requires intentional co-design between educators, researchers, and developers. By focusing on oversight, complementarity, competency, and relational dynamics, we can ensure that AI serves to amplify the human dimensions of education rather than automating them into obsolescence.

Summary updated

Abstract

Human agency is crucial in education and increasingly challenged by the use of generative AI. This meeting report synthesizes interdisciplinary insights and conceptualizes four aspects that delineate human agency: human oversight, AI-human complementarity, AI competencies, and relational emergence. We explore practical dilemmas for protecting and promoting agency, focusing on normative constraints, transparency, and cognitive offloading, and highlight key tensions and implications to inform ethical and effective AI integration in education.
BibTeX
@misc{viberg2026protecting,
  author = {Olga Viberg and Kasia Müldner and René F. Kizilcec and Simon Buckingham Shum and Dorottya Demszky and Dragan Gašević and Ian McInnis and Ioana Jivet and Jelena Jovanovic and Jennifer Meyer and Kou Murayama and Zach A Pardos and Chris Piech and Nikol Rummel and Naomi Winstone},
  title = {{Protecting and Promoting Human Agency in Education in the Age of Artificial Intelligence}},
  year = {2026},
  eprint = {2602.20014},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.20014},
}
Figure