Introduction
Intelligent Tutoring Systems (ITS) generate vast amounts of data on student learning, yet this information often remains trapped in "black boxes," inaccessible to the teachers who need it most. While these systems are effective at tracking step-by-step progress, their rich analytics are typically reserved for backend research or presented directly to students via Open Learner Models. In blended learning environments—where software and human instruction coexist—teachers frequently lack real-time tools to identify which students are struggling and, more importantly, why.
Luna addresses this gap. Developed through a user-centered design process, Luna is a dashboard designed to provide teachers with actionable, real-time insights into student mastery and specific misconceptions, bridging the divide between automated tutoring and classroom instruction.
Methodology
The researchers employed a User-Centered Design (UCD) approach to ensure the dashboard met pedagogical needs rather than simply displaying available data. The process included:
- Contextual Inquiry: Observing and interviewing middle-school teachers to understand their workflows.
- Interpretation & Affinity Diagramming: Organizing user data to identify core requirements.
- Prototyping: Developing a medium-fidelity prototype tested in real classroom settings.
- Integration: Luna was built within the CTAT/TutorShop environment. This framework allows the dashboard to be applied across various tutoring subjects, ensuring scalability beyond a single domain.
The project utilized data from Lynnette, an ITS for basic equation solving, targeting students in grades 6 through 8.
The Luna Dashboard
The design of Luna focuses on three complex data dimensions to support teacher decision-making:
- Class-wide Mastery: Visualizing how many students have mastered specific skills.
- Practice vs. Mastery: Identifying "hard skills" by mapping mastery against the amount of practice required.
- Misconceptions: Tracking the prevalence of specific logic errors across the class.
Key Insights: Misconceptions as the "Killer App"
A primary finding of the study was that teachers found data on student misconceptions (e.g., "subtracting instead of dividing") significantly more actionable than simple mastery scores. Knowing that a student is failing is less helpful than understanding the specific logic error they are making. This allows for targeted interventions that address the root of the student's confusion.
Furthermore, the research identified two distinct usage scenarios for the dashboard:
- Real-Time Decision Support: A "live" view for immediate classroom intervention.
- Exploratory/Reflective Use: Formative reports used outside of class time to plan future lessons.
Conclusion and Future Work
Initial qualitative feedback indicates that Luna's visualization of misconceptions is a highly valuable feature for immediate classroom intervention. By integrating with the CTAT infrastructure, the authors have created a scalable pathway for advanced analytics in diverse tutoring systems.
However, the project remains in its early stages. Future work will focus on transitioning from the medium-fidelity prototype to a fully functioning production version. Additionally, formal studies are needed to quantify how a teacher’s use of Luna directly impacts student learning outcomes across different subjects and tutoring styles.
Summary updated