Distantly Supervised Entity Relation Extraction with Adapted Manual Annotations.

Changzhi Sun, Yuanbin WuAAAI 2019 · OpenReview

Abstract

We investigate the task of distantly supervised joint entity relation extraction. It’s known that training with distant supervision will suffer from noisy samples. To tackle the problem, we propose to adapt a small manually labelled dataset to the large automatically generated dataset. By developing a novel adaptation algorithm, we are able to transfer the high quality but heterogeneous entity relation annotations in a robust and consistent way. Experiments on the benchmark NYT dataset show that our approach significantly outperforms state-of-the-art methods.

Introduction: The Challenge of Noisy Supervision

To build systems that can automatically transform unstructured text into structured knowledge—such as identifying that "Obama" was "born in" the "United States"—large-scale training data is essential. Currently, researchers rely on two primary data sources: manual annotation and distant supervision (DS). While manual annotation provides high-quality labels, it is prohibitively expensive and slow. Conversely, distant supervision automatically aligns external databases (like Freebase) to text, allowing for massive scaling but introducing significant noise. For instance, a sentence mentioning "Obama" and the "United States" might discuss a policy rather than a birthplace, yet DS might incorrectly label it as a "born in" relationship.

Heterogeneity between ACE05 and NYT labeling schemas.

A major hurdle in combining these sources is that manual datasets (e.g., ACE05) and DS datasets (e.g., NYT) are often heterogeneous. As shown in Figure 1, they utilize entirely different categories for entities and relations. Simply merging these datasets often leads to model confusion due to conflicting labeling guidelines. This research proposes a framework to let clean manual data guide noisy DS data without forcing them into a single, rigid schema.

Methodology: Multi-Task Learning and Shared Tasks

The proposed solution is a joint extraction model that handles entities and relations simultaneously using a Multi-Task Learning (MTL) architecture. The framework is designed to capture universal linguistic features while respecting dataset-specific nuances.

Shared Representations and Tasks

The model utilizes a shared biLSTM layer to capture common linguistic features applicable to both datasets. The core innovation lies in identifying "shared tasks" that transcend specific schemas. While entity types might differ (e.g., "PER" vs. "Person"), the boundaries of where an entity starts and ends are generally consistent. The authors introduced two shared tasks:

  1. Entity Span Detection: Identifying the start and end of an entity, regardless of its specific type.
  2. Binary Relation Detection: Identifying if any relationship exists between two entities, regardless of the specific relation type.

Transition Loss Functions

To maintain consistency between the dataset-specific ("private") models and the shared models, the authors introduced a "soft constraint" via transition loss functions. For example, if a private model labels a word as a "Person," the shared model is mathematically constrained to also recognize it as an "Entity." This consistency mechanism makes the model significantly more robust against the false positives inherent in distant supervision.

The proposed multi-task learning architecture with shared tasks and transition losses.

Experimental Setup and Datasets

The approach was evaluated using two benchmark datasets:

  • NYT (New York Times): The primary target dataset, distantly supervised via Freebase, containing approximately 353k training triples with 3 entity types and 24 relation types.
  • ACE05: The "helper" manual dataset, which is small but high-quality, containing 7 entity types and 6 relation types.

Key Results and Insights

The proposed model achieved a new state-of-the-art F1 score of 55.4 on the NYT dataset, a significant improvement over the previous benchmark of 51.7. Several key insights emerged from the ablation studies:

  • Decomposition vs. Merging: Simply merging the datasets ($D^a \cup D^b$) actually decreased performance. Success requires a fine-grained separation of dataset-specific features from universal linguistic features.
  • The "Teacher" Effect: By training on both datasets alternately, the high-quality manual data acts as a regularizer, preventing the model from over-fitting to the noisy samples in the DS data.
  • Schema Invariance: The underlying linguistic structures used to identify entities—such as their appearance after specific prepositions—are invariant across domains, validating the shared task approach.

Limitations and Future Directions

While the model excels at identifying relations between entities in close proximity (within 6 words), performance drops as the distance between related entities increases. Future work may involve incorporating joint decoding algorithms to better capture these long-range dependencies. Additionally, the researchers found that adding more manual data does not always yield linear performance gains, suggesting that Active Learning could be used to identify the most "informative" samples for manual labeling.

Summary updated

Abstract

We investigate the task of distantly supervised joint entity relation extraction. It’s known that training with distant supervision will suffer from noisy samples. To tackle the problem, we propose to adapt a small manually labelled dataset to the large automatically generated dataset. By developing a novel adaptation algorithm, we are able to transfer the high quality but heterogeneous entity relation annotations in a robust and consistent way. Experiments on the benchmark NYT dataset show that our approach significantly outperforms state-ofthe-art methods.
BibTeX
@inproceedings{sun2019distantly,
  author = {Changzhi Sun and Yuanbin Wu},
  title = {{Distantly Supervised Entity Relation Extraction with Adapted Manual Annotations.}},
  booktitle = {AAAI},
  year = {2019},
  url = {https://openreview.net/forum?id=B74bPQGgO6r},
}
Figure