TL;DR
HeriKGQA is a novel question-answering system designed to monitor cultural heritage sites by combining the natural language strengths of Large Language Models (LLMs) with the mathematical precision of Knowledge Graphs. By restricting the LLM to parsing questions and phrasing answers—while offloading all calculations to a deterministic "grounding layer"—it achieves a remarkable 99% accuracy, significantly outperforming standard RAG-based approaches.
What Problem Does This Paper Address?
Cultural heritage sites like the House of Masks in Delos are increasingly monitored by vast sensor networks tracking temperature, humidity, and vibration. While this data is vital for conservation, querying it usually requires database expertise.
LLMs seem like the perfect natural-language interface, but they fail "silently" when faced with raw sensor records. As shown in Figure 1, a standard LLM might interleave readings from two different sensors, misinterpreting a steady state as a rapid temperature jump and raising a false alarm. Furthermore, LLMs are notoriously unreliable at performing complex aggregations (like averages over thousands of rows) or spatial reasoning (like calculating distances between sensor nodes). For a conservator deciding if a monument is at risk, a fluent but mathematically incorrect answer is worse than no answer at all.
Methods
The authors introduce HeriKGQA, which operates on a core principle: the LLM neither computes nor writes the code that computes.
The system architecture, detailed in Figure 2, consists of several key components:
- Sensor Knowledge Graph (KG): Raw data is mapped to a heritage-specific ontology (PANOPTES). This ensures every reading is explicitly linked to its specific sensor, site, and timestamp, preventing the "mixing" errors seen in raw text.
- Semantic Parsing: An LLM maps the user's question into a constrained JSON "intent." This intent specifies filters, aggregations, and target classes without writing free-form code.
- Deterministic Compilation: This intent is automatically compiled into a SPARQL query. Because the compilation is deterministic, it avoids the syntax errors common when LLMs try to write SPARQL directly.
- Grounding Layer: This is the "math engine." It handles spatial distances (using Haversine formulas), temporal facts, and complex anomaly rule evaluations that are difficult to express in pure SPARQL or risky for an LLM to calculate.
- Error Correction: The system uses "intent voting" (running multiple parses and picking the most frequent) and "answer verification" (checking every number in the final answer against the raw data) to ensure total reliability.
Datasets
The researchers evaluated their system using:
- HeriKGQA Benchmark: A new dataset of 500 questions (basic, spatial, temporal, anomaly-related, and follow-up) based on real data from five heritage sites.
- MultiTQ: A general temporal KGQA benchmark used to test the system's portability to other domains.
- Real-World Deployment: 80 questions logged from actual site managers and partners using the ARGUS platform chatbot.
Key Insights and Ideas
- Computation Offloading is Key: The most significant finding is that "grounding"—moving math out of the LLM—is non-negotiable for sensor data. Without the grounding layer, accuracy on anomaly detection questions plummeted from 99% to 66%.
- Metadata Preservation: By using a Knowledge Graph instead of flat text, the system maintains the "provenance" of every data point. This allows for per-sensor rule evaluation even when sensors share the same channel names (e.g., multiple "temp_1" sensors).
- Smaller Backbones Suffice: Interestingly, because the LLM is only used for parsing and verbalization, the system's accuracy remained at 99% even when swapping a massive 120B parameter model for a much smaller 27B parameter one (Qwen-27B).
- Transparency Builds Trust: The system includes an expandable "References" section in its chatbot interface (Figure 3), allowing experts to inspect the exact SPARQL query and retrieved data used to generate an answer.
Key Results
HeriKGQA demonstrated superior performance across all metrics compared to three strong baselines (TextRAG, TableRAG, and Interactive-KBQA):
- Accuracy: Reached 99.0% overall, which is 18.2 points higher than the strongest baseline.
- Speed: It ran nearly 4 times faster than the Interactive-KBQA baseline (12.0s vs 46.7s median latency).
- Reliability: It achieved 100% accuracy on spatial questions, where baselines often failed due to imprecise coordinate arithmetic.
Limitations and Future Work
Despite its success, the authors note several limitations:
- Template Dependency: The primary benchmark uses template-generated questions. While evaluation on real user questions was promising (78.8% accuracy), real-world phrasing is more variable and complex.
- Graph-Specific Logic: The grounding layer currently requires custom code for specific types of spatial or anomaly analysis. Making this layer more "plug-and-play" for new ontologies is a goal for future work.
- Large Result Sets: To avoid errors from sampling, the system currently declines to answer questions that would require processing more than 1,000 rows of data in the final verbalization step.
In conclusion, HeriKGQA provides a robust blueprint for building trustworthy AI interfaces for critical infrastructure and heritage monitoring, proving that for high-stakes data, the best role for an LLM is as a translator, not a calculator.
Summary updated