Evaluating Answers to Definition Questions

Ellen M. VoorheesNAACL 2003

TL;DR

This paper introduces a novel evaluation methodology for question-answering systems that tackle definition-based questions. It proposes a system that combines concept recall with a length-based approximation of precision, demonstrating that this approach correlates well with human judgment of definition quality. This work is crucial for advancing QA systems beyond simple fact retrieval to more nuanced understanding and explanation.

What Problem Does This Paper Address?

Traditional question-answering (QA) systems often excel at answering "closed-class" questions, which have simple, binary right/wrong answers (e.g., "What is the capital of France?"). However, evaluating answers to more complex question types, like those requiring definitions (e.g., "What is mold?" or "Who is Christopher Reeve?"), presents a significant challenge. Unlike binary answers, definitions are subjective and can be expressed in many ways, making it difficult to automatically assess their quality. This paper addresses the need for a robust and scalable evaluation metric for definition-based QA systems, moving beyond simple keyword matching to assess the conceptual completeness and conciseness of generated definitions.

Methods

The authors propose a two-stage evaluation process.

  1. Human Assessment (Initial Round): Human assessors (including the question author and an independent assessor) provide subjective scores for system responses based on content and organization. This initial round helps establish a baseline understanding of human preferences and identify promising systems.
  2. Quantitative Evaluation (Second Round): This is the core of the paper's contribution.
  3. Concept Identification: Assessors identify "information nuggets" or concepts within definitions. These concepts are further categorized as "vital" (essential for a good definition) or "non-vital."
    • Concept Recall: This is calculated as the ratio of vital concepts retrieved by the system to the total number of vital concepts identified by the assessor.
  4. Concept Precision Approximation: Directly measuring precision (correct concepts retrieved / total concepts retrieved) is deemed unworkable due to the difficulty of enumerating all concepts in a response. Instead, the paper borrows from summarization evaluation and uses the length of the response as a proxy for precision. A system is given a "length allowance" (100 non-whitespace characters per retrieved concept). If the response exceeds this allowance, precision is penalized based on the excess length.
  5. F-measure: A combined metric, the F-measure, is used to balance recall and precision. The formula is: $F = \frac{(\beta^2 + 1)RP}{\beta^2P + R}$, where $\beta$ is a parameter controlling the relative importance of recall and precision. The paper primarily uses $\beta=5$, emphasizing recall.

Datasets

The paper describes a pilot evaluation conducted as part of the ARDA AQUAINT program.

  • Benchmark: A set of 25 definition questions was created by NIST assessors. These questions were phrased as "Who is..." or "What is..." followed by the target entity.
  • System Responses: Six AQUAINT contractors submitted eight runs (system outputs) to the pilot.
  • Human Judgments: The evaluation relied on detailed human judgments, including subjective scores and the identification of vital and non-vital concepts for each question.

Key Insights and Ideas

  • Human Agreement on Concepts: Humans show significant agreement on the core concepts that should be included in a definition for a given subject. This agreement is sufficient to reliably compute concept recall.
  • Precision is Tricky: Directly measuring concept precision is extremely difficult and impractical for human assessors.
  • Length as a Precision Proxy: Using response length as an approximation for precision is a pragmatic solution that captures the intuition that shorter, more concise definitions are preferred when they contain the same essential information.
  • F-measure Balances Recall and Conciseness: The F-measure, with an appropriate $\beta$ value, effectively combines concept recall and length-based precision to provide a holistic score that aligns with human judgments of definition quality.
  • Stable Rankings: Despite variations in absolute scores between different human assessors, the relative rankings of systems remained stable, indicating the robustness of the proposed evaluation methodology.

Key Results

  • The proposed evaluation methodology, based on concept recall and length-based precision, produced rankings of systems that correlated well with assessors' subjective opinions on definition quality.
  • System rankings were stable across different assessors, with only minor variations for systems with very close scores.
  • The paper demonstrated that by adjusting the $\beta$ parameter in the F-measure, different user preferences (e.g., prioritizing recall vs. precision) could be accommodated. For instance, increasing the importance of precision (lower $\beta$) led to systems with more concise answers ranking higher.

Limitations and Future Work

  • Crude Precision Approximation: The paper acknowledges that using length as a precision approximation is "crude." While effective, it doesn't capture all nuances of precision.
  • Subjectivity in Concept Identification: While human agreement on concepts is high, there remains an inherent degree of subjectivity in identifying and categorizing concepts as vital or non-vital.
  • Future Work: The authors plan to transfer this evaluation methodology to the larger TREC QA track, indicating its potential for broader adoption and impact in the QA research community. The inclusion of definition questions in the TREC 2003 QA track is a direct outcome of this pilot study.

Summary updated

BibTeX
@inproceedings{voorhees2003evaluating,
  author = {Ellen M. Voorhees},
  title = {{Evaluating Answers to Definition Questions}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2003},
}

Loading related content...

Figure