In 1980, Michael Lebowitz of Yale University introduced the Integrated Partial Parser (IPP), a system that challenged the then-dominant paradigm of exhaustive syntactic parsing in natural language processing (NLP). While contemporary systems often struggled with the computational overhead and fragility of building complete parse trees for every sentence, IPP proposed a "purpose-driven" approach. By focusing exclusively on "interesting" information—specifically international terrorism—IPP demonstrated that a system could achieve robust understanding of complex, real-world newspaper stories by mapping words directly to high-level conceptual structures.
The Problem: Syntactic Overkill
During the late 1970s, NLP research was divided. Systems like SAM and PAM attempted deep, exhaustive analysis but were often slow and easily derailed by ungrammatical text. Conversely, skimming systems like FRUMP were fast but frequently missed critical nuances. Lebowitz observed that human readers do not process all text with equal intensity; a reader skimming a newspaper for specific news has goals that allow them to ignore irrelevant linguistic "fluff." The challenge was to build a parser that could handle the convoluted and often messy language of journalism without getting bogged down in unnecessary syntactic detail.
Methods: Integrated Partial Parsing
IPP’s core philosophy is that high-level semantic expectations should drive the understanding process from the outset. Rather than performing a syntactic parse followed by semantic analysis, IPP integrates these steps.
Script-Based Frameworks
The system relies on "scripts"—predefined knowledge structures representing stereotypical events like $TERRORISM`, `$KIDNAP, or $SHOOT. These scripts provide a template for what information to look for (e.g., actors, victims, weapons).
Top-Down "Requests"
As IPP processes text, it generates "test-action pairs" or requests. For instance, encountering the word "gunmen" triggers a request to identify a subsequent "shooting" verb or a "victim." This top-down expectation guides the bottom-up processing of the text.
Selective Word Processing
To optimize efficiency, IPP categorizes the lexicon into three tiers:
- Immediate Processors: Words that directly build events or tokens (e.g., "exploded," "gunmen").
- Save-for-Later: Modifiers like adjectives or adverbs that are only processed if the noun they modify is deemed important to the script.
- Skippers: Function words and irrelevant terms that are ignored unless specifically needed to fulfill a request.
As shown in the figure above, the output of this process is a hierarchical representation of the event, breaking down a main event into specific "scenes" such as attacks, shootings, and bombings, while identifying the relevant actors and victims.
Key Insights: Interest as a Filter
The most significant contribution of IPP is the use of "interest" as a computational tool. By defining a specific domain of interest, the system can effectively filter out vast amounts of data, making it resilient to the complex or broken syntax often found in news wire services. IPP demonstrates that a formal parse tree is not a prerequisite for high-level understanding; the system jumps directly from lexical items to Conceptual Dependency representations.
Furthermore, IPP employs "plausible inference." If a story mentions both a bombing and a shooting, the system assumes the same actors are involved unless evidence suggests otherwise. This mirrors human skimming behavior, prioritizing speed and "good enough" understanding over perfect accuracy.
Results and Performance
IPP proved remarkably efficient for its time, processing news stories in approximately 2 CPU seconds on a DECsystem 20/50. More importantly, it exhibited robustness that its predecessors lacked, successfully handling convoluted syntax that would typically crash traditional parsers. In qualitative evaluations using real-world stories from the Boston Globe, IPP correctly identified victims, locations, and event types from complex snippets, even when the system had not previously encountered the specific phrasing.
Limitations and Future Directions
Despite its success, IPP was limited by its domain specificity. Expanding the system to new topics required significant manual engineering of new scripts. Additionally, while it excelled at stereotypical events, it struggled with stories requiring deep reasoning about the complex goals and plans of political actors. Lebowitz suggested that future work should focus on expanding vocabulary and developing higher-level knowledge structures to handle a broader range of human narratives beyond international violence.
Summary updated