A Critical Evaluation of Commensurable Abduction Models for Semantic Interpretation

Peter Norvig, Robert WilenskyCOLING 1990

TL;DR

This paper critically evaluates recent abductive models for semantic interpretation, specifically those incorporating "commensurability" to combine different types of evidence. While acknowledging the need for commensurability, the authors argue that current models, like those by Hobbs et al. and Charniak & Goldman, oversimplify by relying on a single scalar measure, leading to limitations in handling ambiguity, representing explanation quality, and modeling complex linguistic phenomena.

What Problem Does This Paper Address?

Language interpretation is the complex task of mapping a string of words into a meaningful representation. This involves integrating evidence from various sources: the lexicon (word meanings), syntax (sentence structure), semantics (meaning of phrases and sentences), and pragmatics (contextual understanding). Often, multiple interpretations are possible for a single sentence. The challenge lies in developing computational models that can weigh these different types of evidence and select the most plausible interpretation. Recent approaches have explored "abduction" – inferring the best explanation for an observation – and a key development has been "commensurability," the idea of representing all evidence in a common currency (like cost or probability) to allow for comparison and combination. This paper critically examines the effectiveness of these commensurable abductive models, particularly focusing on their ability to handle the nuances of natural language understanding.

Methods

The paper evaluates three specific abductive interpretation models:

  • Hobbs et al. (1988): This model frames interpretation as finding the minimal-cost explanation for why a sentence would be true. It uses a logical form annotated with costs for assuming propositions. Rules from a knowledge base can be used to derive propositions, also with associated costs. The goal is to find a set of propositions that entails the logical form with the minimum total cost.
  • Charniak and Goldman (1989a, 1989b): These models shift from a cost-based approach to one based on Bayesian probability theory. They aim to model the probability of events in the real world and use probabilistic inference to determine the most likely interpretation. Later work by Charniak and Goldman introduces a parameter 'E' to account for story coherence.
  • Ng and Mooney (1990): This approach focuses on defining a formal measure of "coherence" to evaluate interpretations. Coherence is defined as maximizing the number of nodes in a proof graph that support multiple observations, normalized to a range of 0-1.

The authors analyze these models through theoretical critique and by applying them to example sentences.

Datasets

Not discussed in the paper. The paper focuses on theoretical evaluation using example sentences rather than empirical evaluation on specific datasets.

Key Insights and Ideas

  • The "Informativeness-Correctness Tradeoff" is Oversimplified: Hobbs et al.'s cost-based model conflates the cost of assuming information with the quality of the explanation. A good explanation might be low-quality but cheap to assume, while a high-quality explanation might be prohibitively expensive. This suggests a need for separate measures for assumption cost and explanation quality.
  • Ambiguity vs. Explanation: Current models struggle to distinguish between true ambiguity (multiple valid interpretations) and the need to find a better explanation. For instance, translating "the pen" as pen(y)$10 conflates the need to find a referent with the need to disambiguate the word "pen."
  • Single Scalar Measures are Insufficient: The core critique is that a single number (cost or probability) cannot adequately capture the multifaceted nature of language interpretation. Factors like word frequency, the quality of an explanation, the need for disambiguation, and the pragmatic context all require distinct representations.
  • Coherence is Important, but Not Everything: While Ng and Mooney's focus on coherence is valuable, their metric has limitations. It can lead to arbitrarily long explanations and doesn't effectively guide search or handle cases where multiple coherent explanations exist.
  • The "Speaker is Always Right" Principle: In communication, the hearer's goal is to understand the speaker's intended meaning, not necessarily to find the most theoretically "optimal" interpretation. This implies a need for models that can quickly find plausible interpretations rather than exhaustively searching for the absolute best.
  • Language is a Complex Activity: The authors argue that current models fail to embrace language as an activity involving actions, goals, beliefs, inferences, and predictions. A more comprehensive theory needs to integrate these "complicated" notions with formal abductive frameworks.

Key Results

The paper does not present quantitative results from experiments. Instead, it offers a qualitative critique of the theoretical underpinnings of the evaluated models. The "results" are the identified limitations and problems with each approach, such as:

  • Hobbs et al.'s model can assign a high cost to a simple noun-noun compound like "lube-oil alarm" due to a lack of specific knowledge, while a low-cost assumption might be more appropriate.
  • Charniak and Goldman's probabilistic models may struggle with texts involving modal verbs, speech acts, or uncooperative speakers. Their coherence metric (parameter 'E') is shown to be insufficient for distinguishing genuine coincidence from narrative coherence.
  • Ng and Mooney's coherence metric can favor overly long explanations and doesn't effectively prioritize between competing interpretations.

Limitations and Future Work

Limitations: * Oversimplification of Evidence: Relying on single scalar measures (cost or probability) fails to capture the distinct nature of different types of evidence and their roles in interpretation. * Handling Ambiguity: Models struggle to differentiate between true ambiguity and the need for a better explanation, and to enforce penalties for contradictions. * Pragmatic and Contextual Understanding: The models do not adequately account for language as a complex activity involving speaker intentions, beliefs, goals, and predictions. * Computational Complexity: Guaranteeing optimal solutions requires exponential time, necessitating approximations for real-time systems. * Deliberate Ambiguity: Models fail to account for intentional ambiguity, such as in puns.

Future Work: The authors suggest that future work should focus on marrying "pre-theoretic 'complicated' notions" (actions, goals, beliefs, inferences, predictions) with the formal apparatus of commensurable abductive interpretation schemes. They also hint at the need for influence diagrams to guide decision-making and for theories of "mention" to complement probabilistic approaches. The ultimate goal is to develop a more robust framework that can handle the full complexity of language use.

Summary updated

BibTeX
@inproceedings{norvig1990critical,
  author = {Peter Norvig and Robert Wilensky},
  title = {{A Critical Evaluation of Commensurable Abduction Models for Semantic Interpretation}},
  booktitle = {International Conference on Computational Linguistics},
  year = {1990},
}
Figure