Abstract
LearnIt is a human-in-the-loop system designed to build custom event-event relation extractors (e.g., "A causes B") in under 20 minutes. By combining traditional bootstrapping with modern embedding-based similarity, it allows researchers to rapidly adapt extraction tools to new domains with minimal manual effort.
Introduction
Extracting relations between real-world events—such as "a flood caused migration"—is vital for situation awareness and automated decision-making. However, building these extractors is traditionally a bottleneck. Developers usually have to choose between two painful options: manually writing exhaustive sets of extraction rules or curating massive labeled datasets to train a classifier.
Furthermore, most existing "rapid customization" tools focus on relations between entities (like "Person X works for Company Y"). Event-event relations are more complex because events can be expressed through various verbs, nominalizations, and complex sentence structures. LearnIt aims to make this process "on-demand," allowing a user to define a new relation type and build a functional extractor for it almost immediately.
System Architecture
The LearnIt system relies on a dual-workflow approach mediated by a user interface, as shown in Figure 1. This architecture balances automated discovery with human oversight to ensure high-quality extraction patterns.
- Workflow 1: Bootstrapping: This follows a classic iterative cycle. Given a few seed event pairs known to have a specific relation, the system searches a large unannotated corpus for instances where those events co-occur. It extracts the patterns connecting them, ranks them, and presents them to a human. The human selects the "good" patterns, which are then used to find even more event pairs.
- Workflow 2: Pattern/Pair Set Expansion: This is the more novel component. The system uses distributional similarity, representing patterns and event pairs as continuous vectors (embeddings). If a user identifies a few high-quality patterns, the system can automatically suggest "mathematically similar" patterns from the embedding space, even if those patterns haven't appeared with the specific seed pairs in the corpus yet.
LearnIt utilizes proposition patterns, which represent the underlying predicate-argument structure (e.g., identifying that "X leads to Y" and "X causes Y" share a similar structural role), rather than simple lexical patterns.
Experimental Setup
The system was developed and evaluated using the following datasets:
- Development Corpus: 1.5 million documents sampled from English Gigaword, serving as the "playground" for the bootstrapping process.
- Embedding Training: To create the similarity vectors, the authors trained a joint embedding model on 13 million
<event1, pattern, event2>triples extracted from the Gigaword corpus. - Test Corpus: 500 documents from Gigaword, exhaustively annotated by humans for six specific causal and temporal relations to evaluate the system's accuracy.
Results and Discussion
The authors tested the system on six relation types: Cause, Preventative, Precondition, Catalyst, Mitigation, and Occurs before. The core innovation is the efficiency: on average, users spent only 18.7 minutes per relation type to build an extractor, resulting in an average of 134 patterns per relation.
As shown in Figure 2, the extractors achieved solid F1 scores across the board. "Cause" relations were extracted with an F1 of 0.73, and "Precondition" reached 0.71. Even the most difficult relation, "Mitigation," achieved an F1 of 0.42, which is significant given the minimal human input. Key insights include:
- Human-in-the-Loop is Essential: Purely automated bootstrapping often suffers from "semantic drift." By keeping a human in the loop to steer the process via a UI, LearnIt prevents the model from picking up irrelevant patterns.
- The Power of Embeddings for Cold Starts: Traditional bootstrapping is limited by what actually appears in a specific corpus. By using embeddings trained on a massive external dataset, the system can suggest synonymous patterns that the user might never have thought of.
Conclusion and Future Work
LearnIt demonstrates that functional event-event relation extractors can be built rapidly without massive labeled datasets. However, the current evaluation required a "trigger word" to be present for a relation to be annotated, which might miss implicit relations. Future work will explore domain-specific embedding retraining for technical fields and extending the system to handle more complex n-ary relations or nested event structures.
Summary updated