TL;DR
This paper introduces the design of a comprehensive lexical database for a German natural language-based expert system. It details the database's structure, the criteria for selecting and coding lexical information, and the tools developed for its construction and maintenance, aiming to bridge the gap between theoretical linguistics and practical NLP applications.
What Problem Does This Paper Address?
Building robust natural language processing (NLP) systems, especially expert systems that interact with users in natural language, requires a deep understanding of words. Traditional dictionaries often lack the detailed syntactic and semantic information needed by computational systems. This paper addresses the challenge of creating a specialized lexical database that goes beyond human-readable definitions to provide the explicit, structured information required for parsing, semantic analysis, and knowledge acquisition in an expert system context, specifically for German.
Methods
The authors propose a relational database approach for their lexical database, leveraging SQL/DS. This allows for efficient storage, retrieval, and management of a large volume of lexical data. Key aspects of their methodology include:
- Structured Lexical Entries: Each word is an entry containing fields for general description, part of speech, morphological codes, and extensive syntactic information (government, complement clauses, adverbials).
- Syntactic Information Coding: A significant focus is placed on defining and coding syntactic features, such as verb valency, obligatory and optional complements, and restrictions on adverbial usage. This is crucial for enabling the natural language analyzer to correctly interpret sentence structures.
- Dictionary Editor: A menu-driven program was developed to facilitate the manual input, updating, and checking of dictionary entries, making the lexicographic process more efficient and less error-prone.
- Standard Database Interfaces: Tools like ISQL and QMF are used for querying, reporting, and extracting data from the database, allowing linguists to analyze and refine the lexical information.
- Concept Hierarchy and Selection Restrictions: For semantic processing, the system utilizes a concept hierarchy and selection restrictions to aid in disambiguation, ellipsis resolution, and anaphoric reference, storing this information separately from the main word relation.
Datasets
The paper does not explicitly mention training or evaluation datasets in the traditional machine learning sense. Instead, it describes the sources used for populating the lexical database:
- Frequency Dictionaries and News Texts: Used to compile a preliminary list of common German words.
- IBM Germany's Dictionary: A pre-existing dictionary of 70,000 entries containing morphological and hyphenation information.
- Legal Domain Texts: Court decisions, accident descriptions from newspapers, and internal IBM legal documents are used to extract vocabulary specific to German traffic law.
- General German Vocabulary: Efforts are made to include the core of general German vocabulary to ensure broader applicability.
Key Insights and Ideas
- The Necessity of Explicit Syntactic Information: The paper strongly emphasizes that for NLP systems, lexical entries must be far more detailed and explicit than for human users. Syntactic features like government and complement types are not optional but essential for computational understanding.
- Relational Databases as a Solution: Using a relational database (SQL/DS) is presented as a superior approach to traditional file organization for managing the complexity and interdependencies of lexical data, offering advantages in control, integrity, and concurrency.
- Bridging Theory and Practice: The project aims to bridge the gap between theoretical linguistic research and practical NLP applications by formulating concrete criteria for coding linguistic phenomena and implementing them in a functional system.
- Challenges in Lexicography: The authors highlight the inherent difficulties in extracting and formalizing linguistic criteria from theoretical works and the practical challenges of ensuring consistency and accuracy in manual coding.
- Semi-Automatic Acquisition Potential: While human intervention is currently required, the paper acknowledges the potential for semi-automatic acquisition of lexical and grammatical information from texts, suggesting future research directions.
Key Results
- Database Design: A structured relational database schema for lexical information is presented, designed to accommodate morphological, syntactic, and eventually semantic data.
- Coding Progress: By November 1985, the project had coded morphological and syntactic information for approximately 5,500 nouns, 3,000 verbs, and 3,500 adjectives.
- Dictionary Size: The target size for the dictionary within the project is 20,000 entries, with a current size of around 12,000 entries at the time of writing.
- Tools Developed: A functional Dictionary Editor and the use of standard database query tools (ISQL, QMF) are established for efficient database management.
Limitations and Future Work
- Semantic Information: Semantic information is not fully coded in the current version of the database, as the authors are still experimenting with different representations.
- Distinguishing Adverbials and Prepositional Objects: The paper notes this as a persistent problem, requiring further investigation into the role and variability of prepositions.
- Semi-Automatic Acquisition: While promising, the fully automatic acquisition of lexical and grammatical information from text is considered doubtful and requires further research.
- Overcoming Discrepancies: The authors are investigating ways to overcome discrepancies between linguistic usage and logical behavior of words.
- Future Work: The immediate next steps include fully integrating the lexical database with the rest of the prototype system, improving concordance programs, and continuing the development of criteria for syntactic subcategorization.
Summary updated