HCRE: LLM-based Hierarchical Classification for Cross-Document Relation Extraction with a Prediction-then-Verification Strategy

Guoqi Ma, Liang Zhang, Hongyao Tu, Hao Fu, Hui Li, Yujie Lin, Longyue Wang, Weihua Luo, Jinsong Su

TL;DR

Cross-document relation extraction is notoriously difficult for Large Language Models (LLMs) because choosing from hundreds of potential relations simultaneously overwhelms their context and reasoning. This paper introduces HCRE, a framework that organizes relations into a hierarchical tree to drastically reduce the options an LLM must consider at each step. To prevent early classification mistakes from derailing the final answer, HCRE employs a novel "prediction-then-verification" strategy that double-checks choices level-by-level, setting a new state-of-the-art for the task.

What Problem Does This Paper Address?

Relation Extraction (RE) is the task of identifying structured relationships between entities in text (e.g., determining that Entity A is the "head of state" of Entity B). While traditional RE focuses on single sentences or documents, cross-document RE requires models to piece together evidence scattered across multiple lengthy texts.

The standard approach relies on a "Small Language Model (SLM) + Classifier" paradigm. Intuitively, one might think replacing the SLM with a powerful LLM would yield an immediate performance boost. However, the authors discovered a surprising bottleneck: LLMs actually perform worse than strong SLMs when forced to select from a massive flat list of predefined relations (such as the 277 relations in the CodRED benchmark).

Stuffing hundreds of options into a prompt causes two issues:

  1. It creates an excessively long context that distracts the LLM from the actual source documents.
  2. It makes it incredibly difficult for the model to distinguish between semantically similar relations.
Comparison between Direct Classification and Hierarchical Classification

Methods

To solve the option-overload problem, the authors developed HCRE (LLM-based Hierarchical Classification for Cross-Document RE), which consists of two main pillars:

1. Hierarchical Relation Tree

Instead of a flat 277-way choice, HCRE uses an advanced LLM (like GPT-4o) to automatically cluster the relations into a top-down taxonomy tree.

  • Level 1: Explicitly separates "valid relations" from "no valid relation" (NA) to handle severe label imbalance.
  • Subsequent Levels: Groups relations by semantic criteria like domain (e.g., politics, finance) and entity types.

By navigating this tree from root to leaf, the LLM only has to choose between an average of 5.5 options at any given node, rather than all 277 at once.

2. Prediction-then-Verification (PtV) Inference Strategy

Hierarchical classification has a fatal flaw: error propagation. If the model makes a mistake at a high level (e.g., picking "finance" instead of "politics"), it can never recover.

To mitigate this, HCRE introduces a level-wise verification loop:

  • Prediction Step: The LLM identifies its top two choices (the best candidate $r_{1st}$ and a suboptimal candidate $r_{2nd}$).
  • Verification Step: The framework generates three alternative "finer-grained" views of the options by expanding $r_{1st}$ and $r_{2nd}$ into their respective child nodes. The LLM is prompted again with these expanded sets. If the new predictions consistently align with the original $r_{1st}$ choice, it passes to the next level. If it fails, the faulty node is eliminated, and the model backtracks.
Overview of the HCRE Framework and PtV Strategy

Datasets

The authors evaluated their approach on two primary benchmarks:

  • CodRED: The primary cross-document RE benchmark. It is evaluated under two settings: closed (where gold text paths containing the evidence are provided) and open (where the model must retrieve the relevant text paths itself).
  • DocRED: A popular document-level RE dataset, utilized specifically to test how well HCRE generalizes across different datasets.

Key Insights and Ideas

  • The Option Bottleneck is Real: The authors explicitly plotted LLM performance against the number of relation options in a prompt. As options drop from 277 down to 8, LLM performance skyrockets, proving that minimizing prompt-option clutter is key to unlocking LLM reasoning in classification tasks.
  • Rethinking RE Metrics: The paper critiques traditional metrics like Maximum F1 and P@K for LLM evaluation. Maximum F1 assumes an unavailable "ideal" threshold, and P@K is highly sensitive to dataset scale. The authors advocate for Micro F1 and Binary F1 as more stable and realistic metrics for generative models.
  • Children as a "Finer Lens": The core intuition behind PtV is that an LLM might confuse two abstract parent concepts, but expanding those concepts into concrete child nodes provides the precise semantic differentiation the model needs to resolve its own ambiguity.

Key Results

HCRE (implemented with a fine-tuned LLaMA-3.1-8B-Instruct backbone) significantly outperformed all baseline models:

  • CodRED (Closed Setting): HCRE achieved a micro F1 of 45.35 and a binary F1 of 58.19, outperforming the strongest SLM baseline (RoBERTa-large + NEPD) by 2.39 and 5.52 points, respectively. It also soundly beat a "Vanilla" directly fine-tuned LLM (38.14 micro F1).
  • CodRED (Open Setting): HCRE achieved 34.91 micro F1, marking a substantial improvement over previous methods.
  • Ablation Impact: Removing the PtV strategy caused the micro F1 to drop sharply from 45.35 to 37.66, proving that the verification step is vital to stopping error propagation.
  • Efficiency Gains: Despite making multiple LLM calls during the verification phase, HCRE reduced the average number of input tokens per instance by 61.6% compared to the Vanilla flat approach, keeping inference latency on par with standard methods.

Limitations and Future Work

While highly effective, the authors note two primary limitations:

  1. Single-Path Constraint: Due to context length limitations, the LLM can only evaluate a single text path at a time during inference, meaning it misses out on cross-path dependencies that could further boost accuracy.
  2. Training Negative Sampling: During the data-reconstruction phase for training the verification step, alternative suboptimal nodes are sampled randomly, which may not perfectly mirror the actual semantic confusions the model encounters during real inference.

For future work, the authors plan to adapt this hierarchical prediction-then-verification framework to other complex information extraction tasks.

Summary updated

Abstract

Cross-document relation extraction (RE) aims to identify relations between the head and tail entities located in different documents. Existing approaches typically adopt the paradigm of ``\textit{Small Language Model (SLM) + Classifier}''. However, the limited language understanding ability of SLMs hinders further improvement of their performance. In this paper, we conduct a preliminary study to explore the performance of Large Language Models (LLMs) in cross-document RE. Despite their extensive parameters, our findings indicate that LLMs do not consistently surpass existing SLMs. Further analysis suggests that the underperformance is largely attributed to the challenges posed by the numerous predefined relations. To overcome this issue, we propose an LLM-based \underline{H}ierarchical \underline{C}lassification model for cross-document \underline{RE} (HCRE), which consists of two core components: 1) an LLM for relation prediction and 2) a \textit{hierarchical relation tree} derived from the predefined relation set. This tree enables the LLM to perform hierarchical classification, where the target relation is inferred level by level. Since the number of child nodes is much smaller than the size of the entire predefined relation set, the hierarchical relation tree significantly reduces the number of relation options that LLM needs to consider during inference. However, hierarchical classification introduces the risk of error propagation across levels. To mitigate this, we propose a \textit{prediction-then-verification} inference strategy that improves prediction reliability through multi-view verification at each level. Extensive experiments show that HCRE outperforms existing baselines, validating its effectiveness.
BibTeX
@inproceedings{ma2026hcre,
  author = {Guoqi Ma and Liang Zhang and Hongyao Tu and Hao Fu and Hui Li and Yujie Lin and Longyue Wang and Weihua Luo and Jinsong Su},
  title = {{HCRE: LLM-based Hierarchical Classification for Cross-Document Relation Extraction with a Prediction-then-Verification Strategy}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1985},
}
Figure