Document-level Relation Extraction (DocRE) is a critical task in natural language processing that involves identifying semantic connections between entities across an entire document. Unlike sentence-level extraction, DocRE requires complex reasoning over multiple sentences and long-range dependencies. Current approaches often face a dilemma: small language models (SLMs) like BERT are efficient but struggle with reasoning, while large language models (LLMs) like GPT-4 offer deep reasoning but are computationally expensive and prone to over-prediction.
This paper introduces the Refiner framework, a hybrid approach that leverages the efficiency of SLMs and the reasoning power of LLMs. By identifying "hard" tasks for the SLM and refining them using an LLM's probabilistic output, the framework achieves state-of-the-art performance with significantly reduced computational overhead.
The Problem: The Tendency Gap
The authors identify a fundamental "tendency gap" between different model architectures. SLMs are generally conservative; they prioritize precision and frequently default to a "No Relation" (NA) classification when faced with ambiguity. This leads to high precision but poor recall. Conversely, LLMs are "over-eager" reasoners. They often hallucinate relations where none exist and struggle with the multi-label nature of DocRE, where a single entity pair might share multiple distinct relations.
As shown in the figure above, SLMs consistently miss more relations than LLMs. However, running an LLM on every possible entity pair in a document is prohibitively slow and expensive. The challenge lies in strategically deploying the LLM only where the SLM is likely to fail.
The Refiner Framework
The proposed Refiner framework operates in three distinct stages to optimize both accuracy and efficiency:
1. Task Distribution
Instead of treating all entity pairs equally, the system uses a dynamic thresholding mechanism. The SLM first processes all pairs. If the SLM's confidence in an "NA" label is only marginally higher than its confidence in a specific relation, the pair is flagged as a "Hard" task. "Easy" tasks (where the SLM is highly confident in NA) are filtered out, while Hard tasks are passed to the LLM for refinement.
2. LLM Refinement with Multiple Choice
For Hard tasks, the LLM is prompted with the document text and a multiple-choice question. The choices include the top-k relations suggested by the SLM. Rather than relying on the LLM's text output, which can be inconsistent, the framework extracts the probability distribution of the first generated token. This treats the LLM as a probabilistic classifier rather than a simple text generator.
3. Probability Fusion and Self-Supervision
To combine the strengths of both models, the framework performs "Probability Balance Estimation." It adjusts the LLM's output variance using a temperature parameter ($\tau$) to align with the SLM's distribution. These distributions are then fused. Finally, a self-supervision step uses the LLM to verify the final predicted relation through a simple True/False prompt, further reducing false positives.
Key Results and Insights
The Refiner framework was evaluated on the DocRED and Re-DocRED benchmarks, yielding several significant findings:
- State-of-the-Art Performance: When applied to top-tier SLMs like DREEAM, the Refiner consistently improved F1 scores by 0.4 to 0.9 points, setting new benchmarks on the Re-DocRED dataset. Compared to zero-shot LLM methods, it improved the F1 score by an average of 25.2%.
- Efficiency Gains: By filtering for Hard tasks, the system reduced the number of entity pairs sent to the LLM. This slashed inference time for a LLaMA3-8B model from 28.9 hours to just 1.21 hours.
- Cost Reduction: For API-based models like GPT-3.5, the Task Distribution module reduced the cost per run from approximately $100 to under $5, making high-quality DocRE accessible for large-scale applications.
Conclusion
The Refiner framework demonstrates that the most effective way to use LLMs in specialized tasks like Relation Extraction is not as a total replacement for smaller models, but as a strategic refiner. By mathematically fusing the probabilistic outputs of both model types, the framework bridges the "tendency gap," achieving superior accuracy while maintaining practical efficiency.
Summary updated