TL;DR
This paper introduces a new taxonomy for classifying dialogue actions in tutorial settings, extending existing frameworks to better capture the specific interactions between students and tutors in mathematical theorem proving. This work is crucial for developing more intelligent and effective AI tutors that can understand and respond appropriately to student and tutor utterances.
What Problem Does This Paper Address?
Intelligent Tutoring Systems (ITS) are increasingly incorporating natural language capabilities to mimic human tutors. However, to build effective dialogue-based ITS, we need a deep understanding of how tutorial dialogues unfold and what "actions" (speech acts or dialogue moves) participants perform. Existing dialogue act taxonomies, like DAMSL, are often too general for the specific nuances of educational interactions. This paper addresses the need for a more specialized taxonomy that can accurately categorize the diverse actions students and tutors take during a learning dialogue, particularly in domains like mathematical theorem proving.
Methods
The authors developed their taxonomy through an iterative process:
- Initial Analysis: They began by analyzing a corpus of tutorial dialogues, using it to verify the general applicability of the DAMSL scheme and identify features specific to tutoring that were missing. This involved defining new move types and drafting annotation guidelines.
- Refinement: The initial taxonomy was applied to a small set of dialogues by the authors themselves. This "preliminary test set" allowed them to refine the taxonomy and its category definitions based on observed data.
- Validation: Finally, a randomly selected subset of dialogues was used as a "validation set" to test the coverage and reliability of the finalized taxonomy.
The core innovation lies in extending the DAMSL framework by adding two new dimensions: Task and Task Progress. The Task dimension focuses on utterances directly related to the learning task, differentiating between actions that contribute to the "solving task" (typically student actions) and those that contribute to the "teaching task" (typically tutor actions). The Task Progress dimension tracks the student's progress and status within the task, such as indicating they are "on-track" or "lost."
Datasets
The research utilized two corpora of tutorial dialogues collected during Wizard-of-Oz experiments in the domain of mathematical theorem proving.
- Corpus-I: Focused on naive set theory.
- Corpus-II: Focused on binary relations.
These corpora were used for both the development and validation of the taxonomy. The authors specifically noted that they avoided dialogues where student utterances contained only formulas to ensure a richer linguistic analysis.
Key Insights and Ideas
- The "Task" Dimension is Crucial for Tutoring: Simply classifying utterances as "assertions" or "requests" isn't enough in a tutoring context. Understanding why an utterance is made (e.g., to solve a step vs. to explain a concept) is key.
- Student and Tutor Roles Matter: Differentiating between student-driven "solving task" actions and tutor-driven "teaching task" actions helps clarify the pedagogical intent behind utterances.
- Tracking Progress is Essential: The "Task Progress" dimension, capturing states like "on-track" or "lost," is vital for an ITS to gauge student understanding and adapt its strategy.
- Nuances of Tutorial Dialogue: Students can be very concise, making it challenging to determine the precise relationship of their utterances to previous discourse, which impacts agreement scores for backward-looking functions.
- Multifunctionality of Utterances: Tutors, in particular, often perform utterances that serve multiple goals simultaneously (e.g., rejecting a step while offering a hint). Multidimensional taxonomies are necessary to capture this.
Key Results
The annotation experiment yielded the following inter-annotator agreement scores (Cohen's kappa):
- Forward Dimension: 0.87 (Very good)
- Backward Dimension: 0.47 (Low)
- Task Dimension: 0.75 (Good)
- Task Progress Dimension: 0.91 (Very good)
The high agreement for Forward and Task Progress dimensions suggests these aspects are well-captured. The lower agreement for the Backward dimension, particularly for categories like "Neutral" and "Step-augmentation," highlights the inherent difficulty in interpreting utterances based solely on previous context in tutorial dialogues. The authors acknowledge that the "Neutral" category, in particular, often served as a catch-all for ambiguous cases.
Limitations and Future Work
- Low Agreement in Backward Dimension: The authors acknowledge the difficulty in reliably annotating backward-looking functions, especially with concise student input. Categories like "Neutral" and "Step-augmentation" showed particular weakness.
- Need for New Categories: The study identified potential needs for new categories, such as a "prompt" type in the forward dimension, a "request next step" in the solving task, and a "check for restart" in task progress.
- Interrelations Between Dimensions: The current presentation doesn't fully capture the complex interrelations between the different dimensions.
- Future Work: The authors plan to conduct a larger-scale annotation to confirm their findings, apply the taxonomy to dialogues in different domains (beyond mathematics), and develop an axiomatic formalization of the dialogue moves for operationalization in plan-based ITS models. They also suggest that the "Task" and "Task Progress" dimensions could be integrated as sub-categories within the "Info-level" dimension of DAMSL, aligning more closely with its original structure.
Summary updated