Abstract
Human agency is crucial in education and increasingly challenged by the use of generative AI. This meeting report synthesizes interdisciplinary insights and conceptualizes four aspects that delineate human agency: human oversight, AI-human complementarity, AI competencies, and relational emergence. We explore practical dilemmas for protecting and promoting agency, focusing on normative constraints, transparency, and cognitive offloading, and highlight key tensions and implications to inform ethical and effective AI integration in education.
Introduction
As generative artificial intelligence (AI) becomes deeply embedded in educational environments, a fundamental question arises: how do we protect human agency—the capacity to act intentionally and exert control over one's environment? The rapid adoption of large language models (LLMs) risks "outsourcing" the very cognitive and metacognitive skills that education is designed to cultivate. If AI handles the drafting, feedback, and assessment, the agency of both students and teachers may be severely diminished.
This paper synthesizes insights from a 2025 interdisciplinary workshop at Marbach Castle, Germany, involving experts from computer science, learning sciences, psychology, and philosophy. It introduces a framework to navigate the "System 0" era—a state where AI acts as an external cognitive layer that processes information before it reaches human consciousness, potentially framing choices and manipulating cognition before critical thinking even begins.
A Four-Dimensional Framework for Agency
To move beyond viewing AI as a mere tool and toward a model of AI as a partner, the authors propose four distinct dimensions of agency:
1. Human Oversight of AI
Agency is maintained only when humans retain the moral and operational authority to override AI systems. This requires "meaningful human control," ensuring that AI actions remain subordinate to human intentions rather than dictating the educational path.
2. Human-AI Complementarity
Rather than a simple division of labor where AI handles "boring" tasks, the report advocates for "Hybrid Adaptivity." This design philosophy ensures AI supports high-order thinking and ethical decision-making, rather than just automating low-level cognitive processes.
3. AI Competencies
Agency is not solely a technical property; it is a human capacity. This involves "AI literacy"—the ability to know when to trust, challenge, or ignore an output—and the intrinsic motivation to remain a reflective agent rather than a passive recipient of AI-generated content.
4. Relational Emergence
Agency does not exist in a vacuum; it emerges from the interaction between individuals, tools, and institutions. This "systemic agency" perspective considers how an AI assistant might alter the dynamics between students, parents, and teachers, shifting the locus of control within the entire educational ecosystem.
Navigating Core Dilemmas
The integration of AI into education presents three central dilemmas that researchers and developers must address:
- Normative Constraints: There is a tension between "hard-coding" educational values into AI and the risk of standardizing thought. While constraints can protect agency by guiding students toward pedagogical goals, they may also create filter bubbles that limit intellectual diversity.
- The Explainability Trade-off: While transparency is a goal, complex models are often inherently opaque. The report suggests a dual approach: continuing the pursuit of explainable AI while simultaneously training humans to be "resilient in ambiguity," learning to work productively with "black box" systems.
- The Zero-Sum Fallacy: The authors challenge the assumption that AI efficiency necessarily reduces human reflection. If AI manages routine administrative or feedback tasks, it may actually free up teacher agency, allowing for deeper, one-on-one mentoring and human connection.
Challenges and Future Directions
The path toward agency-promoting AI faces significant hurdles. Institutional pressures often prioritize cost-cutting and full automation over designs that support human agency. Furthermore, effective human oversight often requires extensive data collection, creating a direct tension with student privacy.
Finally, the report highlights a critical lack of longitudinal data. Most current insights are based on short-term interactions. Future research must investigate how "System 0" affects cognitive development over years, ensuring that we do not inadvertently trade long-term intellectual growth for short-term efficiency.
Conclusion
Safeguarding human agency in the age of AI requires intentional co-design between educators, researchers, and developers. By focusing on oversight, complementarity, competency, and relational dynamics, we can ensure that AI serves to amplify the human dimensions of education rather than automating them into obsolescence.
Summary updated