Abstract
Machine reading in real-world scenarios often involves extracting information from multiple, potentially contradictory, and noisy document sources. Traditional models designed for clean, single-paragraph benchmarks struggle with the inconsistencies found in news clusters or distantly supervised data. We present the RAC-CNN (Represent, Aggregate, and Constrain) architecture, which addresses these challenges by pooling evidence across document clusters and integrating a differentiable factor graph layer to enforce logical consistency. Our approach achieves a 50% relative improvement over previous state-of-the-art methods on the Stanford Plane Crash Dataset.
Introduction: The Challenge of Noisy Information Extraction
Most machine reading research assumes a "perfect information" environment where a single, reliable text contains a clear answer. However, practical information extraction (IE) typically relies on Information Retrieval (IR) to pull a "cluster" of documents that may contain errors, outdated reports, or non-topical mentions. For instance, early news reports of a disaster often provide conflicting casualty counts. Furthermore, many systems utilize distant supervision, training on labels from external databases (like Wikipedia) that may not align perfectly with the retrieved text.
The goal of this work is to bridge the gap between idealized benchmarks and the "real world" by building a reader capable of aggregating evidence across multiple noisy sources while adhering to global "common sense" constraints.
The RAC-CNN Architecture
The RAC-CNN architecture decomposes the machine reading task into three modular components: Representation, Aggregation, and Constraint.
1. Representation and Scoring
The model uses a two-layer Convolutional Neural Network (CNN) to encode the context of every "mention" (candidate values like numbers or names). A slot-specific attention mechanism then scores how well each mention fits a specific "slot" (e.g., Fatalities vs. Passengers).
2. Evidence Aggregation
Rather than selecting the single most confident mention, RAC-CNN aggregates scores across the entire document cluster. This process filters out unique "noise" while rewarding information that is consistently repeated across different articles.
The authors found that Sum-aggregation (pooling evidence) significantly outperforms Max-aggregation (picking the best single mention). They also introduced weighted aggregation based on document metadata, such as topic relevance and publication date, hypothesizing that later reports are generally more accurate.
3. Global Constraints via Differentiable Factor Graphs
To prevent illogical predictions—such as assigning the same number to both Fatalities and Passengers—the model incorporates a factor graph with "Exactly-1" constraints.
Crucially, the authors implement Loopy Belief Propagation (LBP) as a Recurrent Neural Network (RNN). This formulation makes the logical constraints differentiable, allowing the entire system to be trained end-to-end. The model learns to represent mentions in a way that naturally satisfies these global constraints during the forward pass.
Experimental Setup and Results
The architecture was evaluated on the Stanford Plane Crash Dataset, consisting of 80 plane crash events. Each event is associated with a cluster of related news articles, averaging over 2,000 mentions per cluster. Gold labels are derived from Wikipedia infoboxes via distant supervision. A significant challenge in this dataset is the "known unknown" problem: only 33 of the training clusters contain the information needed to fill the Wikipedia slots, requiring the model to decide when not to predict a value.
Key Results
- Performance: RAC-CNN achieved an $F_1$ score of 44.9, a 14.9-point improvement over the previous state-of-the-art.
- Constraint Impact: The addition of the Factor Graph (BP) layer provided a boost of up to 2.8 $F_1$ points.
- Aggregation Power: Moving from mention-level analysis to cluster-level aggregation was the primary driver of performance, confirming that multi-document pooling is essential for noisy IE.
Discussion and Future Work
The study highlights that redundancy is a powerful filter in noisy data. By treating every non-mention word as a potential "null" value, the model effectively handles missing information. Interestingly, the authors found that a single iteration of Belief Propagation was often sufficient; additional iterations could over-constrain the model in rare cases where slots naturally share the same value (e.g., when the number of passengers and fatalities are identical).
Future work includes testing the architecture on larger datasets, improving the handling of textual values (e.g., "no survivors"), and learning aggregation weights directly from data rather than relying on heuristics.
Summary updated