Organizational Issues Arising from the Integration of the Lexicon and Concept Network in a Text Understanding System

Padraig Cunningham, Tony VealeIJCAI 1991

In traditional Natural Language Processing (NLP), a strict separation often exists between the lexicon—containing syntactic information—and the concept network—containing encyclopedic world knowledge. This paper, published in the early 1990s by researchers at the Hitachi Dublin Laboratory and Trinity College Dublin, argues that this "church and state" division leads to significant data redundancy and isolates the parser from crucial conceptual context. To address these issues, the authors introduce TWIG (Text Understanding and Indexing Generation), a system that unifies linguistic and conceptual knowledge into a single, object-oriented representation.

The Unified Representation

The core innovation of TWIG is its integrated knowledge base. Instead of maintaining two separate databases, the system uses a unified IS-A hierarchy where every entry functions as both a concept and a lexeme. For instance, an entry for "Personal Computer" includes syntactic slots for the parser (such as synonyms like "PC" or "Desk Top") alongside conceptual slots (such as the type of processor it uses).

Figure 1: A unified object frame for 'Personal Computer' containing both lexical and conceptual information.

This structure allows the system to generate complex Directed Acyclic Graphs (DAGs) on-the-fly from object frames, optimizing memory usage while providing the parser with rich semantic data. By merging these domains, the system ensures that world knowledge can directly inform the parsing process, leading to more accurate document indexing.

System Architecture and the Blackboard Model

TWIG operates using a Blackboard Architecture, a design where multiple independent agents, or "demons," observe and modify a shared workspace. This workspace is divided into three panels: the Scratchpad for transient data, the Referent Context List for tracking recently mentioned concepts (aiding in anaphora resolution), and the permanent Concept Network.

Figure 2: The blackboard architecture featuring autonomous 'demons' interacting with a shared workspace.

Five autonomous demons drive the understanding process:

  1. Parser: Employs a chart parser and unification grammar to build initial semantic structures.
  2. Concept Formation: Converts these structures into frames (objects).
  3. Anaphoric Resolution: Resolves pronouns by matching new concepts against the context list.
  4. Concept Composition: Links related frames, such as attaching hardware components to a specific computer model.
  5. Filter: Acts as a "librarian," determining which concepts are salient enough for permanent indexing in the network.

Evaluation and Results

The system was evaluated using trade press reviews, specifically a review of the SUN SPARCstation SLC. TWIG successfully demonstrated its ability to learn "unknown" concepts by categorizing the new product, resolving its pronouns, and correctly linking its components into the existing knowledge base.

Key results included:

  • Reduced Complexity: The integration eliminated the duplication of information seen in earlier, bifurcated systems.
  • Parsing Efficiency: The use of a noun phrase pre-processor to "chunk" complex descriptions before they reached the main parser considerably reduced the size of the final chart, making the process faster and more robust.

Conclusion and Future Work

While TWIG successfully reduced architectural complexity and improved concept learning, it remains dependent on manual tagging for "indexable" concepts and requires significant engineering to adapt to new domains. The authors conclude that the system's effectiveness is fundamentally tied to the quality of the integrated network, suggesting that future work should focus on refining this unified structure to further automate the indexing process.

Summary updated

BibTeX
@inproceedings{cunningham1991organizational,
  author = {Padraig Cunningham and Tony Veale},
  title = {{Organizational Issues Arising from the Integration of the Lexicon and Concept Network in a Text Understanding System}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {1991},
}
Figure