Corpus representativeness for syntactic information acquisition

Núria BelACL 2004

TL;DR

This paper argues that for acquiring syntactic information for NLP applications, a small, domain-specific corpus can be more informative than a large, general-purpose one. They introduce a method using cosine distance to measure syntactic behavior divergence and find that domain-tuned corpora, even when smaller, yield more stable and useful results for lexicon induction.

What Problem Does This Paper Address?

Natural Language Processing (NLP) systems, particularly those performing deep parsing, rely heavily on computational lexicons that encode detailed syntactic information about words. However, creating and tuning these lexicons for specific domains (e.g., medical texts, legal documents) is a time-consuming and costly bottleneck for deploying NLP applications. General-purpose lexicons often lack the specificity required for technical texts, leading to parsing errors. The authors aim to address this by investigating how to efficiently acquire domain-specific syntactic information from corpora, focusing on the crucial question of corpus representativeness: what size and composition of a corpus are necessary and sufficient for machine learning techniques to induce this information effectively?

Methods

The core of the paper's methodology revolves around measuring the "syntactic behavior" of words within different domains. They focus on adjectives and define five syntactic contexts (pre-nominal, post-nominal, 'ser' copula, 'estar' copula, and modified by a quantity adverb). For a given word (lemma), they construct a vector representing its occurrences in these contexts within a specific corpus.

To quantify the differences in syntactic behavior between a domain-specific corpus and a general corpus, they employ Cosine Distance (CD). CD measures the divergence between two vectors. A CD closer to 0 indicates significant differences in syntactic patterns, while a CD closer to 1 suggests similarity. This metric is chosen because it captures relative weightings of syntactic features, making it less sensitive to raw frequency differences. They also investigate the relationship between word frequency and CD to ensure that observed differences are not merely artifacts of how often a word appears.

Datasets

The primary dataset used is the CT corpus, a collection of technical specialized texts. This corpus is composed of subcorpora from five different domains: Medicine, Computing, Law, Economy, and Environmental Sciences. Each domain subcorpus ranges from 1 to 3 million words. A "General" subcorpus, primarily consisting of news, is also included for comparison. The authors processed these texts and annotated them with morphosyntactic information. For their experiments, they focused on a list of approximately 300 adjectives that appeared in all corpora.

Key Insights and Ideas

  • Domain-Specificity is Key: The most significant finding is that a relatively small corpus tuned to a specific domain can be more informative for inducing syntactic information than a much larger, general-purpose corpus.
  • Cosine Distance as a Measure of Divergence: Cosine Distance proves to be a robust metric for identifying significant differences in syntactic behavior across domains, even when raw word frequencies vary considerably.
  • Frequency vs. Syntactic Behavior: The study demonstrates that low CD values (indicating significant syntactic divergence) are not directly correlated with simple frequency differences. This suggests that CD captures genuine shifts in how words are used syntactically, rather than just how often they appear.
  • Smaller is Better (with the right focus): The research suggests that for lexicon tuning, the corpus doesn't need to be massive. A well-chosen, domain-representative sample can be sufficient.

Key Results

  • The adjective "paralelo" (parallel) showed a notable difference in its 'ser' copula predicative position when comparing the Computing subcorpus to others, resulting in a lower CD value.
  • Figure 1, plotting CD for 4 subcorpora against the general corpus for about 300 adjectives, shows that more than half of the adjectives have a CD below 0.9, indicating significant domain-specific syntactic variation.
  • Figure 2, which plots CD against the difference in occurrences between two corpora, shows no regular relation, confirming that CD captures genuine syntactic behavior differences, not just frequency effects.
  • Table 3 reveals that for the Computing Corpus, a subcorpus of 600K words already achieved a high CD (mean 0.97, st. deviation 0.009) compared to the full 1.2M word corpus. Surprisingly, the CD for the 3 million word General corpus was lower (0.75) than for the smaller Computing subcorpora.

Limitations and Future Work

The authors acknowledge that their study is part of an ongoing research effort. While they have shown that CD is a good measure for identifying syntactic behavior differences and that domain-specific corpora are valuable, they state that confirming that Machine Learning techniques can truly induce syntactic information from such corpora is part of their future work. They also mention that their current work is limited to Spanish and that the methodology could be extended to other open word categories beyond adjectives.

Summary updated

BibTeX
@inproceedings{bel2004corpus,
  author = {Núria Bel},
  title = {{Corpus representativeness for syntactic information acquisition}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2004},
}
Figure