Abstract
Knowledge Graph (KG) construction traditionally relies on specialized, often brittle, extraction tools. This paper proposes the Auto-KG agent, a framework utilizing Large Language Models (LLMs) as autonomous controllers to orchestrate specialized extraction tools. By leveraging the semantic reasoning capabilities of LLMs like Mistral-7B, the framework addresses common pitfalls in traditional models, such as handling negations and data biases. While quantitative metrics show a "hallucination tax" due to formatting sensitivities, qualitative analysis reveals superior logical consistency and semantic depth compared to standalone extraction models.
Introduction
Knowledge Graphs (KGs) serve as the structural backbone for modern AI, organizing information into semantic triples consisting of a subject, predicate, and object (e.g., Marie Curie -> experimented at -> University of Paris). Traditionally, this extraction process is handled by specialized models like REBEL or KnowGL. However, these tools are frequently "brittle," inheriting biases from their training data and struggling with the complexities of natural language, such as negations and overlapping events. For instance, a sentence like "Fado does not work at IIT" might lead a traditional model to erroneously extract a positive relationship. This research introduces the Auto-KG agent, which positions an LLM as a reasoning "brain" to oversee and refine the outputs of these specialized tools.
Methodology
The Auto-KG framework treats the LLM as an orchestrator rather than a standalone extractor. The architecture consists of three primary components:
- The Brain: An LLM (specifically Mistral-7B or Gemma-2B) acts as the central controller.
- The Toolbox: A database of established extraction models (e.g., REBEL, KnowGL) accessible to the agent.
- The Process: Upon receiving a query, the agent uses system prompts to define its role, triggers appropriate tools, and synthesizes the extracted relations.
To ensure a fair comparison during evaluation, the researchers performed manual mapping, aligning the diverse relation types extracted by tools like REBEL (which may identify dozens of unique types) to the five canonical relations defined in the benchmark dataset.
Experimental Setup
The primary benchmark for evaluation was the CONLL04 dataset, which consists of news articles categorized into four entity types and five specific relation types.
The evaluation focused on the test split, comprising 288 instances, to measure the agent's ability to maintain logical consistency and follow strict output formatting requirements.
Results and Discussion
Semantic Reasoning and Bias Mitigation
Qualitative analysis demonstrated that the Auto-KG agent possesses a significant "semantic understanding" edge. While REBEL often ignored negations, the LLM-based agent correctly identified them, preventing the extraction of false triples. Furthermore, the agent mitigated "location bias"—a common issue where models default to location-based relations due to data overrepresentation. In complex scenarios involving institutional affiliations, the agent successfully identified specific research activities rather than just city-level locations.
The "Hallucination Tax"
Despite superior reasoning, the agent faced challenges with output formatting. Gemma-2B, in particular, failed to return valid JSON in 99 out of 288 responses. Mistral-7B proved more robust, with only 26 such errors. This sensitivity to formatting led to lower F1 scores (0.06 for Mistral vs. 0.16 for standalone REBEL). However, this gap is partly attributed to the "brittleness" of automated metrics; the agent often extracted accurate triples that were absent from the "ground truth" gold standard, resulting in penalties for "extra" correct predictions.
Conclusion and Future Work
The Auto-KG agent demonstrates the potential of LLMs to move beyond pattern matching toward true semantic extraction. Future work will focus on implementing query decomposition to handle complex sentences and a re-ranking module to filter triples. Additionally, the researchers plan to utilize Reinforcement Learning from Human Feedback (RLHF) to fine-tune the agent's tool selection capabilities, further reducing formatting errors and enhancing extraction precision.
Summary updated