Reconciling Adaptivity and Task Orientation in the Student Dashboard of an Intelligent Language Tutoring System

Leona Colling, Tanja Heck, Detmar MeurersACL 2023

TL;DR

This paper presents the design of a novel student dashboard for an intelligent language tutoring system (ILTS) that aims to bridge the gap between adaptive learning and task-oriented instruction in secondary school English classes. The dashboard provides students with transparent insights into their progress and performance while allowing teachers to integrate it seamlessly into their curriculum, addressing the need for both student autonomy and teacher control.

What Problem Does This Paper Address?

Intelligent Tutoring Systems (ITS) are designed to support language learners by providing adaptive practice and insights into their progress. However, designing effective student dashboards for these systems, especially in real-world, blended learning settings like secondary schools, presents several challenges. These include:

  • Reconciling Adaptivity and Task Orientation: Adaptive systems dynamically adjust content, which can conflict with the structured, task-oriented approach often used in language teaching. It's difficult to present a clear learning path that is both personalized and aligned with functional learning goals.
  • Balancing Student Autonomy and Teacher Control: While adaptive systems can foster self-regulation, students (especially younger learners) may struggle with high levels of autonomy. Teachers also need control over curriculum integration, assignment of deadlines, and monitoring student progress.
  • Transparency and User Experience: Presenting complex adaptive information in a way that is understandable and motivating for young learners, who may have developing graphical literacy skills, is crucial.
  • Ecological Validity: Existing ITS dashboards often target higher education and may not adequately address the specific needs of K-12 classrooms, particularly in foreign language learning.

This paper tackles these issues by proposing a co-designed student dashboard that aims to be both adaptive and task-oriented, while also being user-friendly for both students and teachers in a blended learning context.

Methods

The authors propose a conceptual design for a student dashboard that integrates several key features:

  • Task-Oriented Structure: The dashboard is organized around "learning units," each with a clear "functional goal" and "target task." Within these units, "learning targets" (e.g., grammar or vocabulary) are presented, and students work on "realizations" of these targets. This structure explicitly links practice to the overarching communicative goal.
  • Hybrid Navigation: The system offers a blend of learner control and adaptive sequencing. Students can choose to navigate more autonomously or allow the system to guide them through exercises.
  • Progress and Performance Metrics: Instead of relying on traditional Knowledge Tracing (KT) models (which require extensive training data not readily available), the dashboard uses a criterion-referenced approach. It displays progress based on the acquisition of linguistic properties within a realization, visualized through pie charts. Performance is shown as criterion-referenced performance on the most recent ten items, broken down into "solved correctly at first try," "correctly after feedback," and "incorrect or not attempted."
  • Mastery Criterion and Trophies: Students can achieve "mastery" of a realization by successfully completing diagnostic exercises. This mastery is visually represented by a "trophy" symbol. To account for forgetting in adaptive systems, trophies can "gather dust" if mastery is not maintained through practice.
  • Mandatory Exercises: Teachers can designate certain exercises as mandatory, ensuring all students are exposed to specific content. These are integrated into the adaptive sequence but can also be accessed directly by students.
  • Co-design Approach: The dashboard design was developed through an iterative process involving interviews with teachers and a low-fidelity prototype, ensuring it aligns with real-world teaching practices.

Datasets

Not discussed in the paper. The paper focuses on the design and conceptualization of the dashboard, with the implementation based on mock learner data for the prototype.

Key Insights and Ideas

  • Bridging the Gap: The core insight is that adaptivity and task orientation don't have to be mutually exclusive. By structuring the dashboard around functional goals and explicitly linking practice to these goals, the system can provide both personalized learning and a clear sense of purpose.
  • Beyond Traditional KT: The paper highlights the limitations of traditional KT models for progress tracking in scenarios with limited training data and dynamic exercise generation. Their proposed criterion-referenced metric, focusing on linguistic property acquisition and recent performance, offers a practical alternative.
  • Visualizing Mastery and Forgetting: The "trophy" system for mastery, with the added "gathering dust" mechanism for forgetting, is an innovative way to provide students with a tangible and dynamic representation of their learning state, encouraging ongoing engagement.
  • Teacher Empowerment: The inclusion of mandatory exercises and the overall design that respects curriculum alignment empowers teachers to integrate the ITS effectively into their existing teaching practices, rather than seeing it as a separate entity.
  • Focus on "Why": The dashboard design emphasizes making the "why" behind the practice transparent to students, connecting individual exercises to larger learning objectives, which is crucial for motivation and self-regulation.

Key Results

The paper presents a conceptual design and a high-fidelity prototype. While no quantitative results from a deployed system are available yet, the authors report on the co-design process:

  • The design was co-developed with English teachers, ensuring initial validity and alignment with their needs.
  • The prototype was developed using mock learner data.
  • The proposed dashboard features have been evaluated in terms of comprehensibility for seventh graders.

The paper outlines the intended benefits of the design, such as improved student motivation, self-regulation, and better integration into classroom teaching.

Limitations and Future Work

The authors acknowledge several limitations and identify areas for future research:

  • Empirical Validation: The proposed design and concepts require empirical validation through user studies with teachers and students in authentic K-12 settings.
  • Transparency of Mandatory Exercises: While mandatory exercises are crucial for teacher control, their explicit listing might obscure their connection to specific learning targets, potentially increasing cognitive load for students.
  • Global vs. Realization-Specific Diagnostic Exercises: The paper discusses the dilemma of offering a global "Check readiness" button versus linking diagnostic exercises directly to specific realizations, suggesting integration into the adaptive sequence as a potential solution.
  • Performance Visualization Alternatives: The authors acknowledge that their chosen performance visualization (criterion-referenced on recent items) is one of many possibilities and that other aggregations and visualizations could be explored.
  • Customized Learning Units and Mastery: The interaction between teachers customizing learning units and the system's mastery tracking (especially when multiple units cover the same target) presents an open issue.
  • Data Collection for Future Studies: Future work will involve implementing the dashboard using modern libraries and integrating it into the ILTS architecture. A large-scale field study will then be conducted to collect data on learning paths, student characteristics, and the system's practicability and acceptability.

Summary updated

BibTeX
@inproceedings{colling2023reconciling,
  author = {Leona Colling and Tanja Heck and Detmar Meurers},
  title = {{Reconciling Adaptivity and Task Orientation in the Student Dashboard of an Intelligent Language Tutoring System}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure