An Empirical Study on Multiple Information Sources for Zero-Shot Fine-Grained Entity Typing

Yi Chen, Haiyun Jiang, Lemao Liu, Shuming Shi, Chuang Fan, Min Yang, Ruifeng XuEMNLP 2021 · OpenReview

Abstract

Zero-shot fine-grained entity typing (ZFET) aims to predict entity types that were not seen during training. This paper introduces a Multi-Source Fusion (MSF) model that leverages three distinct types of auxiliary information: contextual consistency, type hierarchies, and background knowledge. By intelligently combining these sources, the MSF model achieves significant performance gains, including a 22.84% absolute improvement in Macro F1 on the Wiki dataset compared to previous state-of-the-art methods.

Introduction

Fine-Grained Entity Typing (FET) involves assigning specific labels to entity mentions in text, such as identifying "Apple" as /organization/corporation rather than a generic organization. In dynamic real-world environments, new entity types emerge constantly, making Zero-Shot FET (ZFET) a critical task. ZFET requires models to generalize from seen types to unseen ones without additional labeled training data.

The primary challenge in ZFET is bridging the semantic gap between training and inference types. While various auxiliary information sources—such as Wikipedia descriptions or hierarchical structures—have been used, there has been little systematic study on their relative effectiveness and optimal fusion.

The Multi-Source Fusion (MSF) Framework

The proposed MSF model integrates three specialized modules, each targeting a different information source:

  1. Context-Consistency-Aware (CA) Module: Utilizing a pre-trained BERT model, this module assesses how well a type name fits a given sentence. By masking the entity mention and predicting the probability of a type name in that slot, the model determines contextual consistency.
  2. Type-Hierarchy-Aware (HA) Module: This module employs a Transformer encoder with a masked self-attention mechanism. It constrains types to attend only to themselves and their parent types, ensuring the learned representations respect the underlying ontological hierarchy.
  3. Background-Knowledge-Aware (KA) Module: Treating entity typing as a Natural Language Inference (NLI) task, this module evaluates the "hypothesis" that a type is correct based on three "premises": the context, representative prototypes, and WordNet descriptions. It uses a translation-based loss to align premise and hypothesis representations.
Overview of the Multi-Source Fusion (MSF) architecture and its three auxiliary information modules.

The final prediction is derived from a weighted fusion of the scores generated by these three modules.

Experimental Results

The MSF model was evaluated on two benchmark datasets: BBN (47 types) and Wiki (113 types). The experimental setup involved training on coarse-grained (Level-1) types and testing on unseen fine-grained (Level-2) types.

The MSF model significantly outperformed existing baselines:

  • BBN Dataset: Achieved a 74.02% Macro F1, an 11.42% absolute improvement over the previous state-of-the-art (DZET).
  • Wiki Dataset: Achieved a 57.66% Macro F1, representing a 22.84% absolute gain over the previous best (ZOE).

Ablation studies confirmed that while the KA module is the strongest individual component, the fusion of all three modules is essential for peak performance, demonstrating their complementary nature.

Discussion and Insights

The Role of Context Length

A key finding of the study is the sensitivity of different modules to context length. As shown in Figure 2, the CA module (BERT-based) is particularly effective for long-context scenarios. Because BERT is pre-trained on massive datasets, it possesses inherent knowledge of word-context fit, which is especially useful for "long-tail" unseen types.

Performance (Ma-F1 %) of individual modules and the fused MSF model across varying context lengths.

However, the study also revealed a limitation: while the CA module improves with longer contexts, the HA and KA modules can sometimes introduce noise. In cases with very long contexts (>100 words), the CA module alone can actually outperform the fused MSF model, as seen in the performance drop for MSF relative to CA in the "long" category.

Complementarity and Constraints

The modules offer distinct advantages. The HA module's masked self-attention captures hierarchical dependencies more effectively than simple binary matrices. Meanwhile, the KA module provides external knowledge that helps prevent "common sense" errors. Interestingly, background knowledge often implicitly encodes hierarchy, as descriptions of specific types (e.g., "corporation") naturally overlap with their parent types (e.g., "organization").

Conclusion

The MSF model demonstrates that combining contextual, hierarchical, and external knowledge sources is a powerful strategy for zero-shot entity typing. Future work will explore more dynamic fusion strategies, such as attention-based mechanisms, to better weigh information sources based on input characteristics and further refine the use of pre-trained language models for context consistency.

Summary updated

BibTeX
@inproceedings{chen2021empirical,
  author = {Yi Chen and Haiyun Jiang and Lemao Liu and Shuming Shi and Chuang Fan and Min Yang and Ruifeng Xu},
  title = {{An Empirical Study on Multiple Information Sources for Zero-Shot Fine-Grained Entity Typing}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=nqRDjLsMDK},
}
Figure