TL;DR
This paper introduces a comprehensive framework for "lexical accumulation," encompassing both the acquisition of existing dictionary resources and the production of new lexical sets. The authors propose a flexible, context-free syntax-directed translator and text modification techniques for efficient acquisition, and a formalism for specifying and implementing various production operations, ultimately aiming to build dynamic and combinatorial lexical systems.
What Problem Does This Paper Address?
The creation and management of computational dictionaries and lexicons (referred to as "lexical sets" or LSs) is a notoriously difficult and time-consuming task. While significant effort has been dedicated to acquiring existing digital dictionaries, the production of new lexical resources from existing ones has been less explored, lacking a general computational framework. This paper aims to bridge this gap by providing a unified approach to both acquiring and producing lexical data, and by proposing a novel way to organize and manage these lexical resources within a dynamic system.
Methods
The paper outlines a two-pronged approach:
- Acquisition:
- Syntax-Directed Translator (h-grammar): This method uses a context-free grammar augmented with variables and actions. It allows users to define how source text is parsed and transformed into structured output, offering flexibility in defining output structures that aren't rigidly tied to the source grammar. The output is often represented as CLOS (Common Lisp Object System) objects for easy manipulation.
- Text Modification: Pre-processing and post-processing of source texts are employed to simplify parsing, directly obtain results in simple cases, retain necessary information, and correct common errors. This is often done using regular expressions.
- Production:
- Formalism: The authors propose a formalism for specifying production processes, based on common categories of production and basic operations.
- Categories: These include selection, extraction of substructures, inversion (e.g., English-French to French-English), regrouping (e.g., homographs to polysemes), chaining (e.g., A-B and B-C to A-B-C), and paralleling (e.g., combining multiple English-French dictionaries).
- Operations: Basic operations include assignments, conditionals, string functions, set operations, and advanced transformations like regrouping and splitting.
The paper also introduces the concept of virtual accumulation, where LSs might not be fully available or in a standard form, but can still be used and integrated into a system, with conversion happening on demand. This is complemented by an abstraction of LSs into physical supports, LS supports, lexical sets, and lexical groups, allowing for a modular and dynamic organization of lexical systems.
Datasets
The paper reports acquiring approximately 1.7 million lexical units from about 30 dictionaries of various languages, types, domains, formats, and complexities. Specific examples of acquired resources include:
- DEC, vol. II (Mel'cuk & al 1988)
- French Official Terms (Délégation générale à la langue française)
- Free On-line Dictionary of Computing (D. Howe)
- English-Esperanto (D. Richardson)
- English-UNL (Universal Networking Language)
- I. Kind's BABEL - Glossary of Computer Oriented Abbreviations and Acronyms
The paper also mentions producing about 10 dictionary drafts, totaling around 200,000 entries, as an example of production, such as an English-French-UNL dictionary draft.
Key Insights and Ideas
- Unified Framework for Lexical Accumulation: The core idea is to treat acquisition and production not as separate tasks but as integral parts of a continuous "lexical accumulation" process.
- Flexibility in Acquisition: The h-grammar approach allows users to define custom transformations, moving beyond rigid parsing-to-structure mappings and enabling the retention of specific presentation information when needed.
- Systematic Text Modification: Recognizing the practical utility of text manipulation (beyond just parsing) significantly streamlines the acquisition process.
- Formalism for Production: Providing a structured way to define and implement production operations makes the creation of new lexical resources more systematic and less ad-hoc.
- Virtual Accumulation: This concept is crucial for handling incomplete or dynamically generated lexical resources, enabling more dynamic and potentially larger lexical systems.
- Abstraction Levels for Organization: The proposed hierarchy of physical supports, LS supports, LSs, and lexical groups offers a robust model for organizing and managing complex lexical systems modularly.
Key Results
- Acquisition of approximately 1.7 million lexical units from over 30 diverse dictionaries.
- Production of around 10 dictionary drafts totaling about 200,000 entries.
- Demonstrated efficiency gains in acquisition: for a complex resource (Microsoft Word source files of DEC), human work was reduced by half (1.5 vs. 3 person-months) with a finer analysis and lower error rate compared to previous direct programming methods.
Limitations and Future Work
- Evaluation: The authors acknowledge that a full evaluation of all accumulated lexical data has not yet been performed.
- Future Work:
- Developing methods and environments for testing accumulated lexical data.
- Combining their approach with corpus-based methods.
- Verifying the usefulness and practicality of their lexical system design, particularly the combinatorial and dynamic elaboration of lexical groups and virtual LSs.
- An experiment suggested is building a dictionary server using Internet online dictionaries as resources.
Summary updated