In traditional Natural Language Processing (NLP), a strict separation often exists between the lexicon—containing syntactic information—and the concept network—containing encyclopedic world knowledge. This paper, published in the early 1990s by researchers at the Hitachi Dublin Laboratory and Trinity College Dublin, argues that this "church and state" division leads to significant data redundancy and isolates the parser from crucial conceptual context. To address these issues, the authors introduce TWIG (Text Understanding and Indexing Generation), a system that unifies linguistic and conceptual knowledge into a single, object-oriented representation.
The Unified Representation
The core innovation of TWIG is its integrated knowledge base. Instead of maintaining two separate databases, the system uses a unified IS-A hierarchy where every entry functions as both a concept and a lexeme. For instance, an entry for "Personal Computer" includes syntactic slots for the parser (such as synonyms like "PC" or "Desk Top") alongside conceptual slots (such as the type of processor it uses).
This structure allows the system to generate complex Directed Acyclic Graphs (DAGs) on-the-fly from object frames, optimizing memory usage while providing the parser with rich semantic data. By merging these domains, the system ensures that world knowledge can directly inform the parsing process, leading to more accurate document indexing.
System Architecture and the Blackboard Model
TWIG operates using a Blackboard Architecture, a design where multiple independent agents, or "demons," observe and modify a shared workspace. This workspace is divided into three panels: the Scratchpad for transient data, the Referent Context List for tracking recently mentioned concepts (aiding in anaphora resolution), and the permanent Concept Network.
Five autonomous demons drive the understanding process:
- Parser: Employs a chart parser and unification grammar to build initial semantic structures.
- Concept Formation: Converts these structures into frames (objects).
- Anaphoric Resolution: Resolves pronouns by matching new concepts against the context list.
- Concept Composition: Links related frames, such as attaching hardware components to a specific computer model.
- Filter: Acts as a "librarian," determining which concepts are salient enough for permanent indexing in the network.
Evaluation and Results
The system was evaluated using trade press reviews, specifically a review of the SUN SPARCstation SLC. TWIG successfully demonstrated its ability to learn "unknown" concepts by categorizing the new product, resolving its pronouns, and correctly linking its components into the existing knowledge base.
Key results included:
- Reduced Complexity: The integration eliminated the duplication of information seen in earlier, bifurcated systems.
- Parsing Efficiency: The use of a noun phrase pre-processor to "chunk" complex descriptions before they reached the main parser considerably reduced the size of the final chart, making the process faster and more robust.
Conclusion and Future Work
While TWIG successfully reduced architectural complexity and improved concept learning, it remains dependent on manual tagging for "indexable" concepts and requires significant engineering to adapt to new domains. The authors conclude that the system's effectiveness is fundamentally tied to the quality of the integrated network, suggesting that future work should focus on refining this unified structure to further automate the indexing process.
Summary updated