TL;DR
This paper explores why intelligent tutoring systems (ITS) struggle with adoption among adult learners, despite their proven effectiveness in K-12. By developing and deploying "Apprentice Tutors" and conducting user studies, the authors reveal that sociotechnical factors, rather than just technical limitations, are key barriers. They offer actionable recommendations for designing ITS that better meet the unique needs and motivations of adult learners.
What Problem Does This Paper Address?
Intelligent tutoring systems (ITS) have a strong track record of improving student learning, particularly in traditional K-12 settings. However, their adoption and usage in non-traditional educational contexts, such as workplace training or online adult education, remain significantly underutilized. This paper tackles the question of why this gap exists, hypothesizing that it's not solely due to high development costs, but rather a mismatch between the design of ITS and the specific needs, motivations, and learning preferences of adult learners. Adult learners often juggle multiple responsibilities, are intrinsically motivated, and learn in self-directed environments, presenting unique challenges that traditional ITS may not adequately address.
Methods
The research is structured in two parts:
- Development and Deployment of "Apprentice Tutors": The authors built a web-based ITS called "Apprentice Tutors" designed for college algebra topics. This system incorporates:
- Adaptive Problem Selection: Personalizes the learning experience by presenting problems that target skills a learner hasn't mastered, using Bayesian Knowledge Tracing (BKT) to model skill mastery.
- Real-Time Correctness Feedback: Provides immediate feedback on student input, highlighting correct steps in green and incorrect ones in red.
- Multi-Layer Hinting Strategy: Offers up to three tiers of assistance for each problem step, ranging from highlighting the problematic step to revealing the solution.
- Performance Visualization Dashboards: Allows both students and instructors to track skill mastery and progress.
- The system was deployed in real college algebra classes, with student participation being entirely voluntary.
- Qualitative Investigation of Needs and Challenges: Following the deployment, the researchers conducted five semi-structured focus groups with 42 students and 11 teachers who had interacted with Apprentice Tutors. They used a reflexive thematic analysis framework to identify emergent themes related to user experiences, perceptions, and challenges.
Datasets
The study utilized data collected from the deployment of the "Apprentice Tutors" system in college algebra classes at a technical college. This data included:
- Usage Data: Student access times, interaction logs (button clicks, hint requests, keyboard inputs), and problem completion rates.
- Qualitative Data: Transcripts from focus group sessions with students and teachers.
No new benchmarks were introduced; the focus was on analyzing the adoption and usage patterns of an existing ITS within a specific adult learner demographic.
Key Insights and Ideas
- Sociotechnical Factors Dominate: The primary insight is that the adoption of ITS by adult learners is heavily influenced by sociotechnical factors, not just the technical capabilities of the system. This includes instructor engagement, alignment with teaching practices, and perceived value.
- "Superhuman" Capabilities are Valued: Adult learners appreciate ITS for their "superhuman" aspects, such as providing support when human instructors are unavailable, offering flexibility, and enabling learning anytime, anywhere.
- Hinting and Feedback Need More Depth: Current hint systems are often insufficient, with users wanting clearer explanations of how an answer was reached, not just the answer itself. This is crucial for "getting unstuck."
- Curriculum Alignment is Crucial for Teachers: Teachers need to clearly see how ITS align with their course content and pedagogical goals. A perceived disconnect can lead to frustration and reduced adoption.
- Value Proposition Must Be Clear: For voluntary use, the benefits of ITS must be transparently demonstrated. This includes showcasing progress indicators and the system's effectiveness.
- Onboarding Matters: Streamlining the initial user experience and providing clear tutorials is essential to reduce confusion and frustration, especially for users less familiar with educational technology.
- Human Element is Still Important: While "superhuman" capabilities are valued, concerns were raised that ITS might reduce human interaction, which is still a critical component of learning for many.
Key Results
- Low Initial Adoption, but Retention Exists: Out of 3,510 students with access, only 520 (14.8%) clicked on the tutors at least once. However, among those who did, a significant portion (30.4%) completed at least one problem, and 51.3% of those completed five or more problems.
- Adoption Declines with Scale: The rate of initial tutor clicks decreased significantly as the deployment scale increased from 4 to 58 classes, suggesting challenges in maintaining engagement as the system becomes more widespread.
- Focus Groups Revealed Key Themes: Thematic analysis of focus groups identified five primary categories of insights: perceptions of AI support, need for enhanced explanatory capabilities, pathways for improved tool adoption, usability and value considerations, and suggestions for additional features.
- Specific Needs Identified: Adult learners desire tutors that are easy to understand and navigate, provide explicit links to explanatory content (like videos), and can engage in dialogue-based interactions. Teachers expressed a need for more content and customization options, as well as visibility into student usage and progress.
Limitations and Future Work
Limitations:
- Limited Incentive Structure: The study notes a lack of incentives for teachers to integrate ITS, which can hinder adoption.
- Narrow Course Coverage: The initial deployment covered only a subset of course chapters, limiting the scope of the investigation.
- Lack of Concrete Effectiveness Demonstration: Teachers desired more visual evidence of tutor effectiveness, which was not fully provided in the initial deployment.
Future Work:
- Redesigning Tutors: Future work will focus on redesigning the tutors based on the recommendations, incorporating more instructional materials like textbook resources and lecture videos.
- Expanding Tutor Coverage: The plan is to broaden the tutor's scope to cover more course material.
- Integrating Advanced Analytics: Model tracing will be integrated alongside knowledge tracing and learning analytics visualizations for more robust user insights.
- Longitudinal Studies: The authors aim to conduct longitudinal studies to better understand the long-term impact and scalability of these ITS for adult learners.
Summary updated