Rethinking the Role of LLMs for Document-level Relation Extraction: a Refiner with Task Distribution and Probability Fusion

Fu Zhang, Xinlong Jin, Jingwei Cheng, Hongsen Yu, Huangming XuNAACL 2025 · DOI

Document-level Relation Extraction (DocRE) is a critical task in natural language processing that involves identifying semantic connections between entities across an entire document. Unlike sentence-level extraction, DocRE requires complex reasoning over multiple sentences and long-range dependencies. Current approaches often face a dilemma: small language models (SLMs) like BERT are efficient but struggle with reasoning, while large language models (LLMs) like GPT-4 offer deep reasoning but are computationally expensive and prone to over-prediction.

This paper introduces the Refiner framework, a hybrid approach that leverages the efficiency of SLMs and the reasoning power of LLMs. By identifying "hard" tasks for the SLM and refining them using an LLM's probabilistic output, the framework achieves state-of-the-art performance with significantly reduced computational overhead.

The Problem: The Tendency Gap

The authors identify a fundamental "tendency gap" between different model architectures. SLMs are generally conservative; they prioritize precision and frequently default to a "No Relation" (NA) classification when faced with ambiguity. This leads to high precision but poor recall. Conversely, LLMs are "over-eager" reasoners. They often hallucinate relations where none exist and struggle with the multi-label nature of DocRE, where a single entity pair might share multiple distinct relations.

Figure 3: Comparison of false negatives (predicting "No Relation" when a relation actually exists) between SLMs and LLMs across 100 documents.

As shown in the figure above, SLMs consistently miss more relations than LLMs. However, running an LLM on every possible entity pair in a document is prohibitively slow and expensive. The challenge lies in strategically deploying the LLM only where the SLM is likely to fail.

The Refiner Framework

The proposed Refiner framework operates in three distinct stages to optimize both accuracy and efficiency:

1. Task Distribution

Instead of treating all entity pairs equally, the system uses a dynamic thresholding mechanism. The SLM first processes all pairs. If the SLM's confidence in an "NA" label is only marginally higher than its confidence in a specific relation, the pair is flagged as a "Hard" task. "Easy" tasks (where the SLM is highly confident in NA) are filtered out, while Hard tasks are passed to the LLM for refinement.

2. LLM Refinement with Multiple Choice

For Hard tasks, the LLM is prompted with the document text and a multiple-choice question. The choices include the top-k relations suggested by the SLM. Rather than relying on the LLM's text output, which can be inconsistent, the framework extracts the probability distribution of the first generated token. This treats the LLM as a probabilistic classifier rather than a simple text generator.

3. Probability Fusion and Self-Supervision

To combine the strengths of both models, the framework performs "Probability Balance Estimation." It adjusts the LLM's output variance using a temperature parameter ($\tau$) to align with the SLM's distribution. These distributions are then fused. Finally, a self-supervision step uses the LLM to verify the final predicted relation through a simple True/False prompt, further reducing false positives.

Figure 4: The complete Refiner framework architecture, showing the flow from Task Distribution to LLM Refinement and final Probability Fusion.

Key Results and Insights

The Refiner framework was evaluated on the DocRED and Re-DocRED benchmarks, yielding several significant findings:

  • State-of-the-Art Performance: When applied to top-tier SLMs like DREEAM, the Refiner consistently improved F1 scores by 0.4 to 0.9 points, setting new benchmarks on the Re-DocRED dataset. Compared to zero-shot LLM methods, it improved the F1 score by an average of 25.2%.
  • Efficiency Gains: By filtering for Hard tasks, the system reduced the number of entity pairs sent to the LLM. This slashed inference time for a LLaMA3-8B model from 28.9 hours to just 1.21 hours.
  • Cost Reduction: For API-based models like GPT-3.5, the Task Distribution module reduced the cost per run from approximately $100 to under $5, making high-quality DocRE accessible for large-scale applications.

Conclusion

The Refiner framework demonstrates that the most effective way to use LLMs in specialized tasks like Relation Extraction is not as a total replacement for smaller models, but as a strategic refiner. By mathematically fusing the probabilistic outputs of both model types, the framework bridges the "tendency gap," achieving superior accuracy while maintaining practical efficiency.

Summary updated

BibTeX
@inproceedings{zhang2025rethinking,
  author = {Fu Zhang and Xinlong Jin and Jingwei Cheng and Hongsen Yu and Huangming Xu},
  title = {{Rethinking the Role of LLMs for Document-level Relation Extraction: a Refiner with Task Distribution and Probability Fusion}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.naacl-long.319},
}
Figure