Abstract
The CODER (COmposite Document Expert/extended/effective Retrieval) project investigates the application of artificial intelligence to the retrieval of heterogeneous "composite documents." Central to this effort was the construction of a machine-readable lexicon derived from the Collins Dictionary of the English Language (CDEL). This paper details the conversion of the CDEL into a structured Prolog database, compares its utility against Webster’s Seventh New Collegiate Dictionary, and analyzes the "defining formulas" used to categorize adverbial definitions.
Introduction
In the mid-1980s, information retrieval (IR) systems were largely limited to literal keyword matching. This approach often failed to capture the semantic nuances of language, missing relevant documents that used synonyms or different morphological forms of a search term. To address this, the CODER project at Virginia Tech aimed to integrate Natural Language Processing (NLP) techniques into IR. The cornerstone of this effort was the creation of a comprehensive digital lexicon—a structured repository of word meanings, parts of speech, and semantic relationships.
Methodology: From Typesetter Tapes to Logic Databases
The researchers began with the machine-readable tape of the Collins Dictionary of the English Language (CDEL), originally intended for typesetters. Transforming this raw data into a usable AI resource required a multi-stage pipeline:
- Conversion and Parsing: Using standard UNIX tools such as
lex,yacc, andawk, the team processed approximately 80,000 headwords. The system successfully automated the conversion of 95% of the dictionary. The remaining 5% consisted of highly complex, common words (like the verb "take") whose numerous idiosyncratic senses defied standard parsing rules. - Prolog Integration: The parsed data was structured into a hierarchical Prolog fact base. Prolog was chosen for its ability to represent the dictionary’s inherent hierarchy (Lexeme → Headword → Sense → Sub-sense) as a series of logical facts (e.g.,
word_sense(word, sense_id, ...)) that a reasoning engine could easily traverse.
Comparative Analysis: CDEL vs. W7
A significant portion of the research involved benchmarking the CDEL against the Webster’s Seventh New Collegiate Dictionary (W7), another common machine-readable dictionary of the era. While both dictionaries averaged roughly two definitions per headword, the CDEL proved superior for computational linguistics for several reasons:
- Morphological Richness: CDEL contained significantly more morphological variants (0.33 per headword compared to W7’s 0.14).
- Usage Context: CDEL provided richer usage samples, which are critical for disambiguating word senses in natural language contexts.
The resulting CODER lexicon supported 19 distinct relation types, including morphology (c_MORPH), synonyms (c_COMPARE), and semantic categories (c_CATEGORY).
The Adverb Taxonomy and Defining Formulas
A unique focus of the CODER project was the analysis of adverbs, a category often neglected in lexical research. The researchers discovered that dictionary definitions, while appearing to be free-form text, actually follow rigid "defining formulas." By identifying these patterns, the system could automatically infer the functional type of an adverb:
- Manner: Definitions containing "in a... manner" or "in a... way."
- Time: Definitions containing "at... time" or "at any time."
- Location/Direction: Definitions answering "where?" or "to what place?"
- Degree: Definitions indicating intensity or scale.
This formulaic approach allowed the researchers to map adverb definitions to a functional scheme (Time, Location, Manner, Degree), facilitating more sophisticated semantic analysis during the retrieval process.
Limitations and Future Directions
Despite the success of the conversion, the researchers identified several persistent challenges in 1986:
- Negation Scoping: Determining exactly what a "not" negates within a complex definition remained difficult for automated parsers.
- Intensity Grading: The system struggled to distinguish between fine-grained levels of intensity (e.g., the difference between "accelerated" and "very accelerated").
- Full Parsing: While the adverb grammar was successful, creating a robust parser for every part of speech in the dictionary remained a long-term goal.
Conclusion
The construction of the CODER lexicon represents a foundational step in the evolution of computational linguistics. By treating a traditional dictionary as a semi-structured database and leveraging logic programming, the Virginia Tech team prefigured the development of modern knowledge graphs. Their work on defining formulas demonstrated that the latent structure within human-written definitions could be harnessed to build more intelligent, context-aware information systems.
Summary updated