Hierarchical Modeling of Label Dependency and Label Noise in Fine-grained Entity Typing

Junshuang Wu, Richong Zhang, Yongyi Mao, Masoumeh Soflaei Shahrbabak, Jinpeng Huai

Abstract

Fine-grained Entity Typing (FET) is essential for advanced natural language understanding but faces significant challenges due to complex label hierarchies and structural noise in distantly supervised datasets. This paper introduces FBTree, a framework that jointly models hierarchical label dependencies and structural label noise. By employing a "Forward Tree" (FTree) to predict true entity types and a "Backward Tree" (BTree) to model label corruption, FBTree achieves state-of-the-art performance on major benchmarks like Wiki and BBN.

Introduction

Fine-grained Entity Typing (FET) involves assigning specific, hierarchical labels to entity mentions in text—for instance, identifying "Demi Lovato" as PERSON.ARTIST rather than just PERSON. This specificity is vital for downstream applications like question answering and information extraction.

However, FET is hindered by two primary issues:

  1. Complex Hierarchies: Labels are not independent; they exist in tree structures where capturing nuances between siblings (e.g., ACTOR vs. DIRECTOR) is difficult.
  2. Structural Label Noise: Large-scale datasets often rely on Distant Supervision (DS), which can introduce noise. Existing models often treat this noise as random, failing to recognize that errors typically occur between "nearby" labels in the hierarchy.

Methodology: The FBTree Framework

The proposed FBTree framework addresses these challenges by decoupling the prediction of true types from the modeling of observed noise.

Overview of the FBTree architecture, showing the feature embedding process followed by the Informative Walk (FTree) and the Erroneous Walk (BTree).

Forward Tree (FTree): The Informative Walk

The FTree module models the "true" type distribution $P(Y|x)$. It treats type selection as a random walk starting from the root. At each node, a scoring function—comparing entity features to type embeddings—determines whether to stop or proceed to a child node. To ensure mathematical consistency, "stop nodes" are added to the hierarchy, effectively normalizing it so every valid type terminates at a leaf.

Backward Tree (BTree): The Erroneous Walk

The BTree is the core innovation, modeling how a "true" label $Y$ might be perturbed into an "observed" noisy label $Z$. It simulates a walker that can move backward (up to a parent), across (to a sibling), or remain stationary. This process generates a constrained confusion matrix $P(Z|Y)$. Unlike dense matrices that overfit, BTree’s matrix is strictly governed by the tree's structure, significantly reducing the parameter space.

The model is trained by maximizing the likelihood that the true distribution, when passed through the noise matrix, matches the noisy labels in the training data.

Experiments and Results

The researchers evaluated FBTree on three standard benchmarks: Wiki (FIGER), OntoNotes, and BBN. These datasets vary in size and noise levels (ranging from 24% to 35% noisy mentions).

Performance and Robustness

FBTree consistently outperformed previous state-of-the-art models. On the Wiki dataset, it achieved 70.7% accuracy, and on BBN, it reached 77.3%. A key finding was the model's stability as entity types became more specific.

Comparison of F1 scores across different levels of the hierarchy (Level 1, 2, and 3) for Wiki, OntoNotes, and BBN datasets.

As shown in the figure above, while most models experience a sharp performance drop at deeper levels of the hierarchy, FBTree maintains significantly higher F1 scores at Level 2 and Level 3. Ablation studies further confirmed that removing the BTree component leads to a consistent performance decline, highlighting the necessity of explicit noise modeling.

Discussion and Conclusion

The success of FBTree stems from two key insights:

  • Noise is Structural: Label errors in hierarchical systems are rarely random; they are usually "nearby" in the taxonomy. Modeling noise as a walk on the tree allows the model to effectively "undo" these mistakes.
  • Decoupling Truth from Error: By separating the informative walk from the erroneous walk, the model learns cleaner entity representations even from unreliable training data.

While FBTree is highly effective, it remains dependent on the quality of the pre-defined hierarchy. Future work may explore extending this "Forward-Backward" approach to other hierarchical classification domains, such as image categorization or functional genomics.

Summary updated

Abstract

Fine-grained entity typing (FET) aims to annotate the entity mentions in a sentence with fine-grained type labels. It brings plentiful semantic information for many natural language processing tasks. Existing FET approaches apply hard attention to learn on the noisy labels, and ignore that those noises have structured hierarchical dependency. Despite their successes, these FET models are insufficient in modeling type hierarchy dependencies and handling label noises. In this paper, we directly tackle the structured noisy labels by combining a forward tree module and a backward tree module. Specifically, the forward tree formulates the informative walk that hierarchically represents the type distributions. The backward tree models the erroneous walk that learns the noise confusion matrix. Empirical studies on several benchmark data sets confirm the effectiveness of the proposed framework.
BibTeX
@inproceedings{wu2021hierarchical,
  author = {Junshuang Wu and Richong Zhang and Yongyi Mao and Masoumeh Soflaei Shahrbabak and Jinpeng Huai},
  title = {{Hierarchical Modeling of Label Dependency and Label Noise in Fine-grained Entity Typing}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.24963/ijcai.2021/544},
}
Figure