Represent, Aggregate, and Constrain: A Novel Architecture for Machine Reading from Noisy Sources

Jason Naradowsky, Sebastian RiedelarXiv 2016 · arXiv

Abstract

Machine reading in real-world scenarios often involves extracting information from multiple, potentially contradictory, and noisy document sources. Traditional models designed for clean, single-paragraph benchmarks struggle with the inconsistencies found in news clusters or distantly supervised data. We present the RAC-CNN (Represent, Aggregate, and Constrain) architecture, which addresses these challenges by pooling evidence across document clusters and integrating a differentiable factor graph layer to enforce logical consistency. Our approach achieves a 50% relative improvement over previous state-of-the-art methods on the Stanford Plane Crash Dataset.

Introduction: The Challenge of Noisy Information Extraction

Most machine reading research assumes a "perfect information" environment where a single, reliable text contains a clear answer. However, practical information extraction (IE) typically relies on Information Retrieval (IR) to pull a "cluster" of documents that may contain errors, outdated reports, or non-topical mentions. For instance, early news reports of a disaster often provide conflicting casualty counts. Furthermore, many systems utilize distant supervision, training on labels from external databases (like Wikipedia) that may not align perfectly with the retrieved text.

The goal of this work is to bridge the gap between idealized benchmarks and the "real world" by building a reader capable of aggregating evidence across multiple noisy sources while adhering to global "common sense" constraints.

The RAC-CNN Architecture

The RAC-CNN architecture decomposes the machine reading task into three modular components: Representation, Aggregation, and Constraint.

1. Representation and Scoring

The model uses a two-layer Convolutional Neural Network (CNN) to encode the context of every "mention" (candidate values like numbers or names). A slot-specific attention mechanism then scores how well each mention fits a specific "slot" (e.g., Fatalities vs. Passengers).

2. Evidence Aggregation

Rather than selecting the single most confident mention, RAC-CNN aggregates scores across the entire document cluster. This process filters out unique "noise" while rewarding information that is consistently repeated across different articles.

Overview of the RAC-CNN pipeline: from noisy document clusters to aggregated cluster-level scores.

The authors found that Sum-aggregation (pooling evidence) significantly outperforms Max-aggregation (picking the best single mention). They also introduced weighted aggregation based on document metadata, such as topic relevance and publication date, hypothesizing that later reports are generally more accurate.

3. Global Constraints via Differentiable Factor Graphs

To prevent illogical predictions—such as assigning the same number to both Fatalities and Passengers—the model incorporates a factor graph with "Exactly-1" constraints.

Factor graph constraint inference implemented as an RNN recurrence update.

Crucially, the authors implement Loopy Belief Propagation (LBP) as a Recurrent Neural Network (RNN). This formulation makes the logical constraints differentiable, allowing the entire system to be trained end-to-end. The model learns to represent mentions in a way that naturally satisfies these global constraints during the forward pass.

Experimental Setup and Results

The architecture was evaluated on the Stanford Plane Crash Dataset, consisting of 80 plane crash events. Each event is associated with a cluster of related news articles, averaging over 2,000 mentions per cluster. Gold labels are derived from Wikipedia infoboxes via distant supervision. A significant challenge in this dataset is the "known unknown" problem: only 33 of the training clusters contain the information needed to fill the Wikipedia slots, requiring the model to decide when not to predict a value.

Key Results

  • Performance: RAC-CNN achieved an $F_1$ score of 44.9, a 14.9-point improvement over the previous state-of-the-art.
  • Constraint Impact: The addition of the Factor Graph (BP) layer provided a boost of up to 2.8 $F_1$ points.
  • Aggregation Power: Moving from mention-level analysis to cluster-level aggregation was the primary driver of performance, confirming that multi-document pooling is essential for noisy IE.

Discussion and Future Work

The study highlights that redundancy is a powerful filter in noisy data. By treating every non-mention word as a potential "null" value, the model effectively handles missing information. Interestingly, the authors found that a single iteration of Belief Propagation was often sufficient; additional iterations could over-constrain the model in rare cases where slots naturally share the same value (e.g., when the number of passengers and fatalities are identical).

Future work includes testing the architecture on larger datasets, improving the handling of textual values (e.g., "no survivors"), and learning aggregation weights directly from data rather than relying on heuristics.

Summary updated

Abstract

In order to extract event information from text, a machine reading model must learn to accurately read and interpret the ways in which that information is expressed. But it must also, as the human reader must, aggregate numerous individual value hypotheses into a single coherent global analysis, applying global constraints which reflect prior knowledge of the domain. In this work we focus on the task of extracting plane crash event information from clusters of related news articles whose labels are derived via distant supervision. Unlike previous machine reading work, we assume that while most target values will occur frequently in most clusters, they may also be missing or incorrect. We introduce a novel neural architecture to explicitly model the noisy nature of the data and to deal with these aforementioned learning issues. Our models are trained end-to-end and achieve an improvement of more than 12.1 F$_1$ over previous work, despite using far less linguistic annotation. We apply factor graph constraints to promote more coherent event analyses, with belief propagation inference formulated within the transitions of a recurrent neural network. We show this technique additionally improves maximum F$_1$ by up to 2.8 points, resulting in a relative improvement of $50\%$ over the previous state-of-the-art.
BibTeX
@misc{naradowsky2016represent,
  author = {Jason Naradowsky and Sebastian Riedel},
  title = {{Represent, Aggregate, and Constrain: A Novel Architecture for Machine Reading from Noisy Sources}},
  year = {2016},
  eprint = {1610.09722},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1610.09722},
}
Figure