Abstract
Understanding situated dialogue requires identifying referents in the environment to which the dialogue participants refer. This reference resolution problem, often in a complex environment with high ambiguity, is very challenging. We propose an approach that addresses those challenges by combining learned semantic structure of referring expressions with dialogue history into a ranking-based model. We evaluate the new technique on a corpus of human-human tutorial dialogues for computer programming. The experimental results show a substantial performance improvement over two recent state-of-the-art approaches. The proposed work makes a stride toward automated dialogue in complex problem-solving environments.
Introduction
In situated dialogue, participants interact within a shared physical or digital environment, frequently referring to objects or entities present in that space. For an automated system, such as an intelligent tutoring system for computer programming, resolving these references is a critical prerequisite for effective communication. When a student asks, "Should I use the 2 dimensional array?", the system must identify exactly which variable in the source code the student is discussing.
This task is exceptionally difficult due to three primary factors: environmental ambiguity, where multiple similar objects exist; dynamic environments, where the workspace changes as the user modifies code; and semantic complexity, where referring expressions are structured phrases rather than simple labels. Previous methods often relied on manually curated lexicons or simple word-to-object mappings that fail to capture the compositional nature of human language.
Methodology
The proposed approach utilizes a three-stage pipeline designed to bridge the gap between natural language and digital code objects by leveraging structured learning and temporal context.
Semantic Interpretation
A core contribution of this work is the use of a Linear-chain Conditional Random Field (CRF) to parse the semantic structure of referring expressions. Instead of treating words as independent features, the CRF treats the phrase as a sequence, identifying how words modify one another. For instance, in the phrase "a 2 dimensional array," the model identifies "2" as a numeric value and "dimensional" as an attribute of the "array" category.
Candidate Generation and Ranking
To efficiently manage the search space within a complex programming environment, the system generates a "shortlist" of potential referents. This filtering process uses three distinct information sources:
- Dialogue History: Objects mentioned in recent utterances.
- Task History: Objects that the student has recently edited, created, or interacted with in the IDE.
- Learned Semantics: Objects that possess the specific attributes extracted by the CRF parser.
Finally, a ranking-based classifier—tested across Logistic Regression, Decision Trees, Naive Bayes, and Neural Networks—is used to select the most likely referent from the candidate list. The model integrates features from the dialogue and task history with the semantic attributes to make its final determination.
Experimental Setup
The model was evaluated using the JavaTutor Corpus, a collection of human-human tutorial dialogues where tutors assist students in solving Java programming problems. The full corpus contains 4,857 utterances from 45 tutoring sessions. For this study, a subset of 6 sessions (758 utterances) was manually annotated, identifying 346 referring expressions and linking them to their ground-truth referents in the Java code.
Results and Discussion
The proposed model was compared against two recent state-of-the-art baselines: a lexicon-based ranking model (Iida et al., 2010) and a word-as-classifier model (Kennington & Schlangen, 2015).
As shown in the results, the Neural Network (NN) implementation of the proposed approach achieved an accuracy of 68.5% on expressions containing attributes ($REF_{ATT}$), representing a substantial improvement over the Iida baseline (55.2%) and the Kennington baseline (46.3%).
A key insight from the study is the importance of temporal context. In a coding environment, "my array" typically refers to the object the student just modified. By incorporating task history alongside dialogue history, the model effectively disambiguates referents that might appear identical based on linguistic features alone. Furthermore, experiments using "Gold Standard" (manually perfect) semantic labels showed that accuracy could reach 73.6%, suggesting that as semantic parsing technology matures, reference resolution performance will continue to improve.
Conclusion and Future Work
Despite these gains, a gap remains between the model's performance and human-level agreement, which stands at approximately 85%. This suggests that future models must look beyond immediate history and simple attributes to capture broader discourse intent. Additionally, reducing the reliance on manually labeled data for the CRF parser through unsupervised learning remains a priority for scaling this approach to new domains. By combining structured semantics with environmental context, this work provides a robust framework for reference resolution in complex, real-world problem-solving dialogues.
Summary updated