TL;DR
This paper introduces a context-based strategy for handling pragmatically ill-formed queries in natural language interfaces. Instead of simply rejecting queries that violate real-world knowledge, the system infers the user's intent from the dialogue history and proposes semantically relevant revisions, making interactions more cooperative and robust.
What Problem Does This Paper Address?
Natural language interfaces often struggle with "pragmatic overshoot," where a user's query is syntactically and semantically correct but violates implicit real-world knowledge or the system's understanding of the user's task. For example, asking "Which apartments are for sale?" might be ill-formed if the system's real estate model only allows apartments to be rented, not sold. Human listeners naturally infer the user's likely intent (e.g., looking for apartment buildings or condos that are for sale) and adjust their understanding. This paper addresses the challenge of building a computational system that can perform this kind of cooperative, context-aware correction, moving beyond simple error detection to intelligent response generation.
Methods
The core of the proposed solution is a context-based strategy that leverages a context model of the user's inferred goals and plans. This model is built incrementally from the dialogue history. When pragmatic overshoot is detected, the system employs two main mechanisms:
- Suggestion Mechanism: This component uses heuristics to propose revisions to the ill-formed query. It draws upon the context model to infer what the user likely intended. This involves:
- Simple Substitution Rules: Replacing an attribute, relation, entity set, or function with a semantically similar or contextually appropriate alternative. For instance, if a user asks for the "classification" of a professor, and the system knows the user needs to know the professor's "status" (e.g., full-time, part-time) for an independent study, it suggests "status."
- Expanded Path Rules: Identifying and constructing longer relational paths when the user-specified relation is a "missing logical join." For example, if a user asks for "Dr. Smith's phone number" and the system knows phone numbers are associated with offices, it might infer the user wants the "office phone number" by expanding the path from "Dr. Smith" to "office" to "phone number."
- Selection Mechanism: After generating multiple potential revisions, this mechanism selects the most appropriate one based on three criteria:
- Relevance to the current dialogue: Prioritizing queries that interrogate aspects of the user's current "focused plan" (local context).
- Classification of the revision operation: Categorizing the type of change (e.g., simple substitution, expanded path) and preferring simpler, less drastic alterations.
- Semantic difference: Measuring how semantically close the revised query is to the original, using a generalization hierarchy of attributes and relations.
Datasets
Not discussed in the paper. The system is described as being implemented for a domain consisting of a subset of university courses, requirements, and policies.
Key Insights and Ideas
- Pragmatic Overshoot as a solvable problem: The paper frames pragmatic overshoot not as an unrecoverable error but as an opportunity for cooperative interaction, similar to how humans handle it.
- Context is King: The central idea is that understanding the user's inferred plan and goals (the "context model") is crucial for generating helpful revisions. Without this context, any proposed correction is just a guess.
- Hierarchical Planning for Inference: The use of a "context tree" to represent the user's inferred plan, distinguishing between global and local contexts, allows for more nuanced understanding and prediction of user needs.
- Heuristics for Intelligent Guessing: The suggestion and selection mechanisms provide a structured way to generate and evaluate potential query revisions, balancing semantic similarity with contextual relevance.
- Limiting Expansion: The assumption that relevant path expansions appear on a single path within the inferred plan helps to constrain the search space and avoid generating overly complex or irrelevant suggestions.
Key Results
The paper presents a method and demonstrates its effectiveness through several examples, rather than providing quantitative results from a large-scale evaluation. The examples illustrate how the system can successfully revise queries such as:
- "Which apartments are for sale?" to "Which apartment buildings are for sale?" (relation/entity set substitution).
- "What is the classification of Dr. Smith?" to "What is the status of Dr. Smith?" (attribute substitution).
- "What is Dr. Smith's phone number?" to "What is Dr. Smith's office phone number?" (expanded relational path).
- "When does Mitchel meet?" to "When does Mitchel's section meet?" (expanded relational path).
The authors argue that their approach leads to more cooperative and informative responses than simply rejecting ill-formed queries.
Limitations and Future Work
The authors acknowledge several limitations and suggest future research directions:
- Domain Specificity: The current implementation is for a specific university domain.
- Path Expansion Constraints: The assumption that relevant path expansions appear on a single path within the inferred plan might miss some associations (e.g., between a person's SSN and their name). Further investigation is needed to produce precisely the set of meaningful and relevant path expansions.
- Handling More Than Two Entity Sets: Extensions are needed to handle relationships among more than two entity sets in path expansions.
- Other Classes of Pragmatic Overshoot: The current work primarily focuses on one class of pragmatic overshoot. Future work should extend the techniques to handle other types, such as queries that are irrelevant to the user's underlying task.
- Exceeding World Model Knowledge: The system assumes it can provide the information needed. Future work should address detecting and responding to queries that exceed the knowledge represented in the underlying world model.
Summary updated