TL;DR
Cross-document relation extraction is notoriously difficult for Large Language Models (LLMs) because choosing from hundreds of potential relations simultaneously overwhelms their context and reasoning. This paper introduces HCRE, a framework that organizes relations into a hierarchical tree to drastically reduce the options an LLM must consider at each step. To prevent early classification mistakes from derailing the final answer, HCRE employs a novel "prediction-then-verification" strategy that double-checks choices level-by-level, setting a new state-of-the-art for the task.
What Problem Does This Paper Address?
Relation Extraction (RE) is the task of identifying structured relationships between entities in text (e.g., determining that Entity A is the "head of state" of Entity B). While traditional RE focuses on single sentences or documents, cross-document RE requires models to piece together evidence scattered across multiple lengthy texts.
The standard approach relies on a "Small Language Model (SLM) + Classifier" paradigm. Intuitively, one might think replacing the SLM with a powerful LLM would yield an immediate performance boost. However, the authors discovered a surprising bottleneck: LLMs actually perform worse than strong SLMs when forced to select from a massive flat list of predefined relations (such as the 277 relations in the CodRED benchmark).
Stuffing hundreds of options into a prompt causes two issues:
- It creates an excessively long context that distracts the LLM from the actual source documents.
- It makes it incredibly difficult for the model to distinguish between semantically similar relations.
Methods
To solve the option-overload problem, the authors developed HCRE (LLM-based Hierarchical Classification for Cross-Document RE), which consists of two main pillars:
1. Hierarchical Relation Tree
Instead of a flat 277-way choice, HCRE uses an advanced LLM (like GPT-4o) to automatically cluster the relations into a top-down taxonomy tree.
- Level 1: Explicitly separates "valid relations" from "no valid relation" (NA) to handle severe label imbalance.
- Subsequent Levels: Groups relations by semantic criteria like domain (e.g., politics, finance) and entity types.
By navigating this tree from root to leaf, the LLM only has to choose between an average of 5.5 options at any given node, rather than all 277 at once.
2. Prediction-then-Verification (PtV) Inference Strategy
Hierarchical classification has a fatal flaw: error propagation. If the model makes a mistake at a high level (e.g., picking "finance" instead of "politics"), it can never recover.
To mitigate this, HCRE introduces a level-wise verification loop:
- Prediction Step: The LLM identifies its top two choices (the best candidate $r_{1st}$ and a suboptimal candidate $r_{2nd}$).
- Verification Step: The framework generates three alternative "finer-grained" views of the options by expanding $r_{1st}$ and $r_{2nd}$ into their respective child nodes. The LLM is prompted again with these expanded sets. If the new predictions consistently align with the original $r_{1st}$ choice, it passes to the next level. If it fails, the faulty node is eliminated, and the model backtracks.
Datasets
The authors evaluated their approach on two primary benchmarks:
- CodRED: The primary cross-document RE benchmark. It is evaluated under two settings: closed (where gold text paths containing the evidence are provided) and open (where the model must retrieve the relevant text paths itself).
- DocRED: A popular document-level RE dataset, utilized specifically to test how well HCRE generalizes across different datasets.
Key Insights and Ideas
- The Option Bottleneck is Real: The authors explicitly plotted LLM performance against the number of relation options in a prompt. As options drop from 277 down to 8, LLM performance skyrockets, proving that minimizing prompt-option clutter is key to unlocking LLM reasoning in classification tasks.
- Rethinking RE Metrics: The paper critiques traditional metrics like Maximum F1 and P@K for LLM evaluation. Maximum F1 assumes an unavailable "ideal" threshold, and P@K is highly sensitive to dataset scale. The authors advocate for Micro F1 and Binary F1 as more stable and realistic metrics for generative models.
- Children as a "Finer Lens": The core intuition behind PtV is that an LLM might confuse two abstract parent concepts, but expanding those concepts into concrete child nodes provides the precise semantic differentiation the model needs to resolve its own ambiguity.
Key Results
HCRE (implemented with a fine-tuned LLaMA-3.1-8B-Instruct backbone) significantly outperformed all baseline models:
- CodRED (Closed Setting): HCRE achieved a micro F1 of 45.35 and a binary F1 of 58.19, outperforming the strongest SLM baseline (RoBERTa-large + NEPD) by 2.39 and 5.52 points, respectively. It also soundly beat a "Vanilla" directly fine-tuned LLM (38.14 micro F1).
- CodRED (Open Setting): HCRE achieved 34.91 micro F1, marking a substantial improvement over previous methods.
- Ablation Impact: Removing the PtV strategy caused the micro F1 to drop sharply from 45.35 to 37.66, proving that the verification step is vital to stopping error propagation.
- Efficiency Gains: Despite making multiple LLM calls during the verification phase, HCRE reduced the average number of input tokens per instance by 61.6% compared to the Vanilla flat approach, keeping inference latency on par with standard methods.
Limitations and Future Work
While highly effective, the authors note two primary limitations:
- Single-Path Constraint: Due to context length limitations, the LLM can only evaluate a single text path at a time during inference, meaning it misses out on cross-path dependencies that could further boost accuracy.
- Training Negative Sampling: During the data-reconstruction phase for training the verification step, alternative suboptimal nodes are sampled randomly, which may not perfectly mirror the actual semantic confusions the model encounters during real inference.
For future work, the authors plan to adapt this hierarchical prediction-then-verification framework to other complex information extraction tasks.
Summary updated