TL;DR
This paper introduces a novel system that bridges the gap between traditional keyword-based Information Retrieval (IR) and more sophisticated Information Extraction (IE). It allows users to construct complex, on-the-fly queries using not just keywords but also linguistic and positional constraints, offering more precise and conveniently formatted results than standard IR systems, without the high customization cost of traditional IE.
What Problem Does This Paper Address?
The digital information landscape is exploding, making it increasingly difficult for users to find specific, relevant information. Traditional Information Retrieval (IR) systems, which typically return lists of document URLs based on keyword searches, often lack precision and provide output that isn't ideal for many tasks. For instance, finding all companies associated with a specific person might be better served by an alphabetically ordered list of companies with sentence-level context, rather than just a list of documents.
On the other hand, Information Extraction (IE) systems can provide highly structured and precise results (like tables or database records) and allow for more complex searches based on grammatical relationships and positions of keywords. However, current IE systems are usually programmed beforehand, making them suitable only for repetitive tasks like processing newsfeeds. Customizing an IE system for a new domain is an expensive and time-consuming process, requiring domain experts, linguists, and significant manual effort to define new vocabulary, grammatical structures, and extraction patterns. This paper aims to democratize IE by enabling end-users to create sophisticated queries interactively, without needing to program patterns.
Methods
The core of the system is built upon the existing "Highlight" IE engine, which uses shallow syntactic processing. The key innovation lies in the development of a user interface that allows users to construct queries using a combination of:
- Keywords: Standard search terms.
- Sortal Constraints: Specifying the type of entity (e.g., "person name," "company name," "location").
- Linguistic Constraints: Specifying grammatical relationships or features (e.g., "syntax group = noun phrase").
- Positional Constraints: Defining the relative positions of search items (e.g., "immediately precedes," "same sentence").
These constraints are encoded using a distributed representation, which allows for flexible and partial semantic analyses. The system preprocesses documents to create an index of linguistic information, enabling faster query processing. At query time, the system uses an IR component to filter documents based on non-optional keywords or sortal information from a summary file, before applying the more complex IE constraints. This approach aims to be more intuitive than full natural language querying but offers significantly more power than basic keyword search.
Datasets
The paper mentions using the Wall Street Journal (WSJ) and Financial Times (FT) datasets for evaluating performance. These datasets consist of news articles. The specific number of articles used for testing are:
- WSJ: 100 articles
- FT: 60 articles
Key Insights and Ideas
- Bridging the IR/IE Divide: The system successfully demonstrates that it's possible to offer IE-like precision and output formatting through an IR-like user interface, making advanced extraction accessible to a wider audience.
- On-the-Fly Querying for IE: The ability to construct complex IE queries interactively, without pre-programming patterns, is a significant step towards making IE more flexible and user-friendly for one-off tasks.
- Distributed Representation for Flexibility: Using distributed representations for linguistic analysis allows the system to handle both shallow and deep semantic structures, enabling queries to be as specific or general as needed.
- Preprocessing for Speed: Preprocessing documents significantly reduces query processing time, making the system practical for real-world applications, though it increases storage requirements.
- IR as a Filter: Integrating an IR stage before the IE component can dramatically speed up processing by filtering out irrelevant documents early on, especially for queries with specific keywords.
Key Results
The paper presents performance figures demonstrating the system's speed. Key results include:
- Preprocessing Speedup: Adding preprocessing reduced total processing time by approximately 75% for the tested queries.
- IR Stage Speedup: The IR stage further reduced processing time by a few percent up to 10% for the tested datasets.
- Query Times:
- A query for "Esso and a location" (Figure 6) completed in 0.5 seconds, as "Esso" was a strong discriminator.
- A query for "Person, X of Company" (Figure 7) took 27 seconds.
- A query for protein interactions (Figure 8) took 66 seconds, due to a higher number of potential hits.
- Qualitative Results: The paper showcases example queries and their results (Figures 3-8), illustrating the system's ability to extract structured information like company names, positions, and protein interactions, presented in tables.
Limitations and Future Work
The authors acknowledge several limitations and suggest future directions:
- User Expertise Levels: While the system offers novice and expert modes, the expert mode's interface could be further refined.
- "Blocking Tactics": The original Highlight system used "blocking tactics" (preventing one rule from firing if another did). The current customisable version needs to introduce a similar mechanism for choosing between alternative queries to achieve equivalent functionality.
- External Program Calls: The original Highlight system could call external programs, a feature not yet replicated in the customisable version.
- Scalability and Optimization: While preprocessing improves speed, the authors note that further optimization of preprocessed file storage and encoding schemes is possible. They also plan to investigate further ways to improve constraint satisfaction speed.
- Full Natural Language Querying: The system is not a full natural language querying system, which remains a more ambitious goal.
- Evaluation: A large-scale evaluation of the user-customisable version is still pending, though the underlying engine has been evaluated previously.
Summary updated