A Method to Reduce Large Number of Concordances

Maria Pozzi, Javier Becerra, Jaime Rangel, Luis Fernando LaraCOLING 1980

TL;DR

This paper introduces a novel computational method to efficiently analyze large sets of word concordances, a crucial but time-consuming task in lexicography. By employing a tree-based structure to group similar contextual usages of a word, the method significantly reduces the number of concordances that need manual review, preserving essential linguistic information.

What Problem Does This Paper Address?

Defining words accurately in a dictionary requires a thorough understanding of their usage, which is typically derived from analyzing numerous examples of the word in context – known as concordances. However, for common words, the sheer volume of concordances can be overwhelming, making manual analysis impractical and prone to missing subtle nuances. Existing methods of simply taking a random sample risk losing valuable grammatical and semantic information. This paper addresses the challenge of efficiently extracting maximum linguistic insight from a massive corpus of word occurrences while minimizing the manual effort required.

Methods

The core of the proposed method involves a two-stage process:

  1. Contextual Grouping (Tree Generation): For a given target word "W," the algorithm first extracts a window of up to four words to the left and four words to the right of "W" from each concordance. Crucially, it also considers the grammatical category of these surrounding words. This information is used to build a tree structure. The root of the tree is the word "W" itself. Direct descendants of a node are words that appear immediately adjacent in the context and share the same grammatical category. This process recursively builds a tree that represents the contextual relationships of words surrounding "W."
  2. Concordance Reduction: Once the tree is constructed, a reduction algorithm is applied. This algorithm traverses the tree, prioritizing paths with higher frequencies (indicating more repeated contexts). Based on the level in the tree and the frequency of a particular context, a predetermined formula is used to calculate the number of concordances (Q) to retain from a total of F occurrences. The goal is to select a representative subset that captures the essential meanings and uses without losing critical information.

Datasets

The study utilizes the "Corpus del Español Mexicano Contemporáneo (CEMC)," a large corpus containing 1,973,151 occurrences. A preliminary step involved automatically labeling each word in the corpus with its grammatical category.

Key Insights and Ideas

  • Contextual Similarity is Key: The paper highlights that grouping concordances based not just on surrounding words but also their grammatical categories is crucial for identifying similar linguistic usages.
  • Tree Structure for Organization: The use of a tree structure effectively organizes the complex relationships between words in context, making the analysis more systematic.
  • Frequency-Based Reduction with Contextual Awareness: The reduction algorithm intelligently balances the need to reduce the number of concordances with the preservation of information, using frequency and tree depth as key indicators.
  • Empirical Determination of Reduction Formulas: The authors emphasize that the specific formulas for reducing concordances were empirically derived through extensive trials by linguists, suggesting a practical, data-driven approach.
  • Beyond Lexicography: The method has a broader application in identifying recurring linguistic patterns and expressions, useful for various natural language processing tasks.

Key Results

The initial results were highly encouraging. For words with a medium number of concordances (up to 600), the method achieved a reduction of 30% to 40% without reported loss of information. The authors anticipate even greater efficiency for words with higher frequencies. The computational process of generating trees is noted as time-consuming, but ongoing work aims to optimize this.

Limitations and Future Work

The primary limitation acknowledged is the computational time required for generating the tree structures, especially for words with very high frequencies. The authors are actively working on optimizing this aspect of the system. Future work will likely focus on further refining the reduction algorithms and exploring the application of this method to other linguistic analysis tasks.

Summary updated

BibTeX
@inproceedings{pozzi1980method,
  author = {Maria Pozzi and Javier Becerra and Jaime Rangel and Luis Fernando Lara},
  title = {{A Method to Reduce Large Number of Concordances}},
  booktitle = {International Conference on Computational Linguistics},
  year = {1980},
}
Figure