Problem Localization Strategies for Pramatics Processing in Natural-Language Front Ends

Lance A. Ramshaw, Ralph M. WeischedelACL 1984

TL;DR

This paper tackles the challenge of explaining why a natural language system fails to fulfill a user's request, even when the request is syntactically and semantically sound. The authors propose "problem localization" strategies, primarily using planning trees and "surprise scores," to identify the most significant reason for failure, enabling more helpful and informative system responses.

What Problem Does This Paper Address?

Imagine you're using a natural language interface to a system like Unix, and you ask it to do something. The system understands what you're saying, but it can't actually do it. This isn't a misunderstanding of your words; it's a "pragmatic overshoot" – the request, while understandable, is beyond the system's capabilities or the current state of the world. For example, you might ask to print a file, but the printer is offline, or you don't have permission to read the file.

When these failures happen, simply saying "I can't do that" is unhelpful. The real challenge is to pinpoint why the system failed and communicate that reason effectively to the user. This is crucial for building cooperative and user-friendly systems. The paper focuses on identifying the most significant failure within a complex set of conditions that led to the overall failure, rather than just listing all possible reasons.

Methods

The core of the paper's approach lies in using a planner to verify the "presumptions" within a user's request. These presumptions are treated as subgoals in a plan to achieve the user's ultimate goal. The planner attempts to build a plan; if it fails, the paper proposes methods to analyze the failed plan structure to find the root cause.

The authors introduce two main strategies for "problem localization":

  1. Statistical Ranking Using Surprise Scores: This method relies on dynamically collected statistics about how often specific subgoals (nodes in the planning tree) succeed or fail. A "success ratio" is calculated for each node.
  2. Surprise Score: A score is derived from the success ratio. A successful node that rarely succeeds gets a high surprise score (indicating something unusual happened). A failed node that often succeeds gets a strongly negative surprise score (indicating a significant, surprising failure).
  3. Heuristics: * AND nodes: At AND nodes (where all subgoals must succeed), the strategy is to report the most critical failure – the one least likely to be changed by the user. This often corresponds to the node with the most negative surprise score.
  4. OR nodes: At OR nodes (where only one subgoal needs to succeed), the strategy is to report the most hopeful failure – the one the user is most likely to be able to affect or repair. This also tends to be the node with the most negative surprise score.
  5. Special-Purpose Problem Localization Rules: To add more nuanced knowledge beyond statistics, the authors propose condition-action rules. These rules can examine the success or failure of other nodes in the planning tree or consider the broader planning context.
  6. Example: A rule might state that if a critical child of an AND node fails (e.g., IS-OWNER fails for a file), then the surprise score of a parent node (like READ-PERM) should be suppressed, effectively making another branch more likely to be reported. Conversely, a successful child might enhance a parent's score.

Datasets

Not discussed in the paper. The examples are drawn from a simulated Unix environment for a pragmatic overshoot component of a natural language interface.

Key Insights and Ideas

  • Duality of AND/OR Node Heuristics: There's an interesting duality: at AND nodes, we want to report the most critical failure (least user-controllable), while at OR nodes, we want to report the most hopeful failure (most user-controllable). Surprisingly, both often align with identifying the most surprising failure (most negative surprise score).
  • "Surprise" as a Proxy for Significance: The core idea is that surprising failures (nodes that usually succeed but failed) or surprising successes (nodes that usually fail but succeeded) are good indicators of where the real problem lies.
  • Beyond Simple Failure Reporting: The paper moves beyond just saying "failure occurred" to actively diagnosing which failure is most important to communicate. This is key for user interaction.
  • The "Why Not?" Question: The strategies developed are directly applicable to answering a user's "Why not?" question, explaining why a goal couldn't be achieved.

Key Results

The paper doesn't present quantitative results from a large-scale evaluation. Instead, it demonstrates the effectiveness of its heuristics and strategies through detailed examples using a PRINT-FILE command in a Unix context. The examples illustrate how surprise scores and special-purpose rules can correctly identify the most significant failure among multiple possibilities. For instance, in one example, the failure of IS-TEXT-FILE is identified as the most critical failure for PRINT-FILE because it's something the user cannot directly influence, unlike file permissions.

Limitations and Future Work

  • Level of Detail: The authors acknowledge that the leaves of the planning tree might be too detailed for a user. Future work should address selecting an appropriate level of detail for responses, potentially by modeling user expertise.
  • World Knowledge: Surprise scores are a "rough guide" and may not capture true significance if it depends on external world knowledge not present in the system (e.g., a printer being down for days vs. hours). Special-purpose rules are proposed to mitigate this.
  • Other Applications: The authors suggest that problem localization strategies could be valuable in other areas like expert systems (for "Why?" and "Why not?" questions), backward-chaining inference, and top-down parsing (e.g., in ATN grammars).
  • Recovery Strategies: Further work is needed to integrate problem localization with actual recovery strategies, if any are appropriate.
  • Formulaic Approaches: The paper briefly mentions the possibility of a more complex formula to combine surprise scores, but deems it harder to control.

Summary updated

BibTeX
@inproceedings{ramshaw1984problem,
  author = {Lance A. Ramshaw and Ralph M. Weischedel},
  title = {{Problem Localization Strategies for Pramatics Processing in Natural-Language Front Ends}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {1984},
}
Figure