Developing a Teacher Dashboard For Use with Intelligent Tutoring Systems

Vincent Aleven, Franceska Xhakaj, Kenneth Holstein, Bruce M. McLarenIWTA@EC-TEL 2016 · OpenReview

Introduction

Intelligent Tutoring Systems (ITSs) have proven highly effective at providing personalized, step-by-step guidance to students. However, these systems often operate as "black boxes" for the teachers overseeing the classroom. While an ITS generates vast amounts of data regarding student knowledge, affect, and "wheel-spinning" (persistent struggling without progress), this information is rarely presented in a way that is easy for K-12 teachers to digest or act upon. Researchers at Carnegie Mellon University are addressing this gap by developing a teacher dashboard designed to transform high-resolution ITS data into actionable classroom instruction.

Methodology: User-Centered Design

The research team employed a multi-stage User-Centered Design (UCD) approach to ensure the dashboard met the practical needs of middle school teachers. This process began with 11.5 hours of contextual inquiry, including interviews and observations with six teachers to understand their current data usage.

To explore potential features, the researchers used a technique called "Speed Dating with Storyboards." They presented 22 different scenarios to teachers to gauge interest in features like automated student grouping or alerts for "gaming the system."

A storyboard illustrating a "next-day" planning scenario where a teacher uses dashboard data to adjust her lecture.

Following these initial phases, the team iterated from medium-fidelity mockups to a high-fidelity dashboard built in Tableau. This prototype was then evaluated in a controlled classroom study where teachers used real data from their own students to plan lessons.

Key Insights and Dashboard Design

The study revealed several critical requirements for educational dashboards in "blended" learning environments:

  1. The Actionability of Errors: Teachers were less concerned with simple completion metrics and more interested in why students were failing. In algebra, identifying specific misconceptions allows a teacher to immediately pivot to a targeted "mini-lesson."
  2. The Trust vs. Summary Trade-off: While teachers appreciated high-level summaries, they expressed a need to "drill down" into raw student steps to verify the system's conclusions.
  3. Teacher Agency: The dashboard is viewed as a support tool for teacher expertise, not an automated replacement for instructional decision-making.

The researchers identified two primary use cases: "on-the-spot" real-time triage during computer lab sessions, and "next-day" reflective planning used outside of class to adjust future instruction.

The high-fidelity dashboard prototype, integrating skill mastery bars, student progress scatter plots, and misconception heatmaps.

Results and Future Directions

Preliminary qualitative findings indicate that ITS-specific metrics, such as learning curves and wheel-spinning indicators, provide insights that traditional observation cannot capture. Access to the dashboard appeared to shift instructional focus toward common misconceptions identified by the software rather than general progress.

While the quantitative impact on student learning gains is still under analysis, the study highlights the necessity of specialized UI designs for different teaching scenarios. Future work will focus on fully integrating the dashboard into the live CTAT/Tutorshop backend to provide real-time analytics and optimizing the interface for mobile devices to support active classroom management.

Summary updated

BibTeX
@inproceedings{aleven2016developing,
  author = {Vincent Aleven and Franceska Xhakaj and Kenneth Holstein and Bruce M. McLaren},
  title = {{Developing a Teacher Dashboard For Use with Intelligent Tutoring Systems}},
  booktitle = {IWTA@EC-TEL},
  year = {2016},
  url = {https://openreview.net/forum?id=NU2iKgg908},
}
Figure