Using Linguistic Resources to Evaluate the Quality of Annotated Corpora

Max SilberzteinCOLING 2018

TL;DR

This paper investigates the quality of annotated corpora, which are crucial for modern NLP. It reveals that even large corpora like the Open American National Corpus have significant vocabulary coverage and tagging issues, leading to unreliable NLP analyses. The authors propose using handcrafted linguistic resources to improve corpus quality and NLP model performance.

What Problem Does This Paper Address?

Modern Natural Language Processing (NLP) applications, especially those using statistical and neural network methods, heavily rely on large, pre-annotated corpora (like the Penn Treebank or the Open American National Corpus) for training and analysis. These methods essentially compare new text to a reference corpus to understand word meanings and grammatical structures. The core assumption is that these reference corpora are complete and reliable. However, this paper questions that assumption, investigating whether existing corpora are truly sufficient for robust NLP. The authors aim to understand the minimum corpus size needed for stable vocabulary coverage, the reliability of the stored information, and the impact of missing information on analysis quality.

Methods

The paper employs a series of experiments using the NooJ linguistic development environment to analyze the "Slate" corpus, a sub-corpus of the Open American National Corpus.

  • Vocabulary Stability: The Slate corpus was split into two halves (even and odd numbered files) to assess vocabulary overlap. Differences in wordforms between these subsets indicate vocabulary instability.
  • Vocabulary Evolution: The paper tracks the growth of the vocabulary (both wordforms and unique lexemes) as the corpus size increases, visualized in a graph.
  • Linguistic Resource Application: The DELAS English dictionary and a phrasal verb dictionary were applied to the Slate corpus to evaluate coverage of standard English vocabulary and phrasal verbs.
  • Compound Word and Phrasal Verb Analysis: The paper specifically examines how compound words and phrasal verbs are tagged and represented in the corpus, comparing them to dictionary entries.
  • Hapax Legomena (Hapaxes) Analysis: The frequency and impact of words appearing only once (hapaxes) in the corpus are analyzed, particularly their effect on statistical parsers.
  • Tagging Accuracy: A systematic evaluation of the accuracy of the Penn tag set annotations within the Slate corpus was performed using NooJ.

Datasets

  • Slate Corpus: A sub-corpus of the Open American National Corpus (OANC), consisting of 4,531 articles and 4,302,120 wordforms, tagged with the Penn tag set.
  • DELAS Dictionary: Used to evaluate the coverage of standard English vocabulary.
  • NooJ's Phrasal Verb Dictionary: Contains 1,260 phrasal verbs, used to assess phrasal verb coverage in the Slate corpus.

Key Insights and Ideas

  • Vocabulary Instability: Even a 4-million-wordform corpus like Slate exhibits significant vocabulary instability. Roughly half of the wordforms appear in only one of the two randomly split sub-corpora, suggesting the corpus is too small to have a stabilized vocabulary.
  • Limited Coverage of Standard Vocabulary: Applying the DELAS dictionary revealed that the Slate corpus covers only about one-sixth of standard English vocabulary. To achieve 50% coverage of standard vocabulary, an estimated 16 million additional wordforms would be needed.
  • Inconsistent Compound Word Tagging: The corpus struggles with compound words and discontinuous expressions. Many compounds are treated as sequences of individual words, and there's confusion between hyphenated compounds and sequences containing hyphens or em dashes. This leads to incorrect tagging for a significant portion of wordforms.
  • Underrepresentation of Phrasal Verbs: Less than one-third of the phrasal verbs listed in a dedicated dictionary had any occurrences in the Slate corpus, indicating poor coverage of these crucial linguistic units.
  • Hapaxes Pollute Analysis: A substantial portion (one-third) of the Slate corpus vocabulary consists of hapaxes. Statistical parsers that don't handle hapaxes appropriately will produce unreliable results.
  • Significant Tagging Errors: At least 25% of the wordform tags in the Slate corpus were found to be incorrect. Systematic errors, like confusing Wh-determiners with pronouns or misclassifying wordforms based on punctuation, suggest the use of flawed automatic disambiguation rules during annotation.
  • Granularity Matters: NLP applications need more than just part-of-speech tags. Distinguishing between types of nouns (human, concrete, abstract) and verbs (transitive, intransitive) is crucial for accurate analysis, but current corpora lack this fine-grained information.

Key Results

  • Vocabulary Instability: Approximately 50% of wordforms in the Slate corpus appeared in only one of two randomly split sub-corpora.
  • Standard Vocabulary Coverage: The Slate corpus covers only about 1/6th of the standard English vocabulary as represented by the DELAS dictionary.
  • Compound Word Coverage: Only 10% of the English vocabulary consists of compound words, and the Slate corpus covers only 10% of these. To achieve 50% coverage of compounds, over 32 million wordforms would need to be added.
  • Phrasal Verb Coverage: Less than 1/3 of phrasal verbs from a dictionary were found in the Slate corpus.
  • Hapax Frequency: 31,275 out of 88,945 wordforms in the Slate corpus are hapaxes (approximately 35%).
  • Tagging Accuracy: Over 20% of non-proper name wordforms in the Slate corpus were considered incorrect by NooJ. At least 25% of occurrences of a specific wordform were found to have incorrect tags.

Limitations and Future Work

  • Limitations:
    • The study highlights that even large corpora are not "complete" and suffer from significant vocabulary gaps and annotation errors.
  • The analysis of tagging errors suggests systematic issues, possibly due to automatic disambiguation rules that are not transparent or easily correctable.
  • The paper focuses on the Slate corpus, and while it's part of a larger standard corpus, the findings might not generalize perfectly to all annotated corpora.
  • Future Work/Proposals:
    • Compare with Dictionaries: Compare tagged wordforms against standard English dictionaries (like DELAS) to correct typos and "impossible" tags.
  • Tag Compound and Discontinuous Expressions: Develop methods to tag compound words and discontinuous expressions as single units, using existing dictionaries.
  • Develop Local Grammars: Create simple local grammars to correct tags that are incompatible with English grammar, potentially replacing flawed automatic disambiguation algorithms.
  • Enhance Tag Sets: Enrich the tag set with more detailed linguistic information (e.g., noun types, verb complements) to create a "semantic-rich" tagged corpus for more sophisticated NLP applications like Information Extraction and Machine Translation.

Summary updated

BibTeX
@inproceedings{silberztein2018using,
  author = {Max Silberztein},
  title = {{Using Linguistic Resources to Evaluate the Quality of Annotated Corpora}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
}
Figure