A Methodology for Automatic Identification of Nocuous Ambiguity

Hui Yang, Anne de Roeck, Alistair Willis, Bashar NuseibehCOLING 2010

TL;DR

This paper introduces a machine learning-based methodology to automatically identify "nocuous ambiguity" – linguistic expressions that are likely to cause misunderstandings among different readers. By combining various linguistic heuristics and human judgment data, their approach aims to flag potentially problematic ambiguities in technical texts, improving clarity and reducing errors in fields like requirements engineering.

What Problem Does This Paper Address?

Ambiguity in language is a well-known challenge, but not all ambiguities are created equal. Some are harmless or even intended, while others can lead to significant misunderstandings and costly errors, particularly in technical domains like software requirements engineering. Traditional approaches often try to resolve ambiguity by finding a single "correct" interpretation. However, this paper argues that sometimes, the problem isn't the ambiguity itself, but the risk of different people interpreting it differently. They define "nocuous ambiguity" as ambiguity that leads to diverging interpretations among stakeholders. The challenge is to automatically identify these nocuous ambiguities without requiring manual review of every potentially ambiguous sentence.

Methods

The core of the methodology is a machine learning (ML) architecture that learns to predict nocuous ambiguity. This involves several key steps:

  • Human Judgments: The process starts by collecting human interpretations of ambiguous sentences. Participants are asked to provide their preferred reading or label it as ambiguous. This forms the ground truth.
  • Ambiguity Threshold: A crucial concept is the "ambiguity threshold" (τ). This threshold determines how much consensus among readers is needed for an ambiguity to be considered "innocuous." If the certainty of any single interpretation falls below this threshold, the ambiguity is flagged as potentially nocuous. Different application domains might require different thresholds (e.g., safety-critical systems need a lower threshold).
  • Heuristics: The system uses a set of "heuristics" – rules or indicators derived from linguistic theory and statistical analysis – that predict factors favoring one interpretation over another. For coordination ambiguity, these include factors like word similarity, collocation frequency, and morphological agreement. For anaphora ambiguity, heuristics focus on linguistic properties, context, and statistical information to identify likely antecedents.
  • Machine Learning: The outputs of these heuristics are fed into an ML classifier. The paper explores various ML algorithms (like LogitBoost and Naive Bayes) to learn from the human-judged data and predict whether a new instance of ambiguity is nocuous or innocuous based on the heuristic scores.

Datasets

The paper utilizes datasets of human judgments collected for two types of ambiguity:

  • Coordination Ambiguity: A dataset of 138 coordination instances gathered from requirement documents, previously used by Chantree et al. (2006).
  • Anaphora Ambiguity: A dataset of 200 anaphora instances collected from requirements documents, specifically focusing on third-person pronouns.

These datasets were used for training the ML models and for evaluation. The paper does not introduce new benchmark datasets but leverages existing ones to validate their methodology.

Key Insights and Ideas

  • Nocuousness as a Property of Stakeholders: The paper emphasizes that ambiguity isn't just a property of text but of text in relation to a set of stakeholders. This shifts the focus from resolving ambiguity to understanding its potential impact on communication.
  • The Ambiguity Threshold is Key: The concept of an ambiguity threshold (τ) is central. It allows for flexible tolerance of ambiguity, acknowledging that different contexts demand different levels of certainty. The paper shows how varying this threshold significantly impacts the classification of nocuous ambiguity.
  • Heuristics Work Best in Concert: Individual heuristics have limited predictive power. The ML approach effectively combines these heuristics to capture complex patterns and dependencies, leading to more robust predictions.
  • Coordination vs. Anaphora Risk: The results suggest that, in technical documents, anaphora ambiguity might be less likely to lead to misunderstandings than coordination ambiguity, at least based on their datasets and methodology.
  • Weak Thresholds for Anaphora: For anaphora ambiguity, using "weak thresholds" (Wy and Wn) for positive and negative antecedent predictions helped improve recall by catching more potential nocuous cases, even at the cost of slightly lower precision (more false positives).

Key Results

  • Coordination Ambiguity: The proposed ML model (CM-1) achieved an accuracy of over 75% on average. It showed a significant improvement of approximately 21% in accuracy compared to a previous model (CM-2) and naive baselines, particularly in the mid-range ambiguity threshold levels (0.45-0.75). This improvement is attributed to extended heuristics and the LogitBoost algorithm.
  • Anaphora Ambiguity: The ML models (AM-1 and AM-2) consistently improved upon baseline accuracy at higher ambiguity thresholds (τ≥0.75). While performance dipped at lower thresholds, the models showed an average increase of about 3.4% on F-measure compared to a naive nocuous baseline (BL-2), indicating better overall performance in identifying nocuous anaphora. The model with weak thresholds (AM-1) showed a higher F-measure, suggesting it effectively catches more nocuous instances.

Limitations and Future Work

  • Context Dependency: A significant limitation acknowledged is that human judgments were collected without surrounding context. In reality, discourse focus and contextual information heavily influence interpretation.
  • Reader Competence and Domain Knowledge: Factors like a reader's language competence, familiarity with a domain, and shared tacit knowledge can affect how ambiguity is perceived, which is not fully captured.
  • Heuristic Tailoring: The methodology relies on tailoring ML algorithms to specific heuristic types (e.g., LogitBoost for numeric, Naive Bayes for Boolean). This suggests a need for careful management of interdependent components.
  • Future Work: The authors suggest exploring more complex ambiguity types (like PP attachment and quantifier scope), investigating whether these are amenable to a heuristics-based approach, and further refining ambiguity tolerance thresholds. They also aim to improve system performance and validate its practical use.

Summary updated

BibTeX
@inproceedings{yang2010methodology,
  author = {Hui Yang and Anne de Roeck and Alistair Willis and Bashar Nuseibeh},
  title = {{A Methodology for Automatic Identification of Nocuous Ambiguity}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2010},
}

Loading related content...

Figure