Abstract: Farah Ftouhi, Russel Dsouza, Lance Calvin Lim Gamboa, Asim Abbas, Mubashir Ali, Yue Feng, Mark G. Lee, Venelin Kovatchev. Proceedings of the Eighth Fact Extraction and VERification Workshop (FEVER).
2025.
Introduction
In an era of rapid information spread, manual fact-checking cannot keep pace with the volume of daily content. Automated systems are essential, but verifying a claim is more complex than a simple search; it requires understanding nuances, retrieving specific evidence from massive knowledge bases, and reasoning through potentially conflicting information. The "OldJoe" system was developed for the AVeriTeC Shared Task to address these challenges by verifying real-world claims against a structured knowledge store using a sophisticated Retrieval-Augmented Generation (RAG) pipeline.
Methodology: The Five-Phase Pipeline
The authors propose a five-stage pipeline centered on In-Context Learning (ICL) and efficient database operations. The system decomposes complex claims into smaller, verifiable parts to mimic a human fact-checker's workflow.
- Evidence Embedding: Evidence documents are semantically split into 2048-character chunks and vectorized using the
jina-embeddings-v3model. These are stored in a PostgreSQL database using thepgvectorextension. - Question & Query Generation: The system uses Qwen3-14B to generate four targeted investigative questions and corresponding search queries for each claim.
- Hybrid Retrieval & Re-ranking: A hybrid search is performed within a single SQL query, combining semantic similarity (vector search) and keyword matching (BM25) using Reciprocal Rank Fusion (RRF).
- Answer Generation: For each question, the LLM generates a concise answer based strictly on the retrieved evidence, citing sources appropriately.
- Veracity Judgment: The LLM reviews the original claim alongside the generated question-answer pairs to assign one of four labels: Supported, Refuted, Conflicting, or Not Enough Evidence.
Hybrid Retrieval and SQL Integration
A significant technical insight of this work is the implementation of retrieval and RRF re-ranking directly in SQL. By leveraging PostgreSQL and pgvector, the team avoided complex external orchestration layers. This approach proved highly effective; using the Prometheus evaluation metric (an LLM-as-a-judge framework), the PostgreSQL RRF approach scored 3.80, significantly outperforming standalone FAISS (1.20) or BM25 (2.90).
Key Results
In internal development tests, the qwen3-14b-awq model was the top performer, achieving a veracity label accuracy of 0.494 and an AVeriTeC score of 0.492. The use of AWQ-quantized versions allowed the 14B parameter model to run within a 24GB VRAM envelope, making the system accessible on mid-range hardware. On the official test set, the system achieved an overall AVeriTeC score of 0.151, reflecting the high difficulty of unseen, real-world data.
Conclusion and Future Work
OldJoe demonstrates that claim decomposition and SQL-native hybrid search are powerful tools for automated fact-checking. Future work will focus on fine-tuning models specifically on verification datasets, exploring larger model variants (72B+ parameters) for stronger reasoning, and improving generalization techniques to handle the diversity of real-world claims more effectively.
Summary updated