Introduction
As Large Language Models (LLMs) transition from general-purpose chatbots to specialized industrial tools, their ability to process high-stakes, domain-specific data becomes critical. In public transport management, for instance, an AI must accurately interpret "where" and "when" incidents occur to assist dispatchers effectively. However, current benchmarks often rely on synthetic or English-centric data, failing to capture the nuances of real-world industrial legacy systems. This paper introduces a novel benchmark to quantify hallucinations in LLMs when processing complex spatio-temporal traffic data.
The H&PS Traffic Incidents Dataset
The cornerstone of this research is the H&PS Traffic Incidents Spatio-Temporal Dataset, a massive collection of 99,869 unique records from Vienna, Austria, covering the decade between 2013 and 2023. Unlike standard academic benchmarks, this dataset reflects the messy reality of industrial data, featuring German-language descriptions of 14 disruption categories, such as "Acute Track Damage" or "Illegal Parking."
Methodology and Hypotheses
The researchers evaluated nine leading LLMs, including GPT-4, Llama-3, and Claude-3. To identify ways to improve performance, they tested three preprocessing hypotheses designed to "nudge" models toward higher accuracy:
- Sentence Indexing: Numbering report sentences to aid information tracking.
- Natural Language Timestamps: Converting raw data into human-readable dates and times.
- Cross-Lingual Translation: Translating German reports into English to leverage the models' primary training language.
Performance was analyzed using Multiple Linear Regression (MLR) to isolate the impact of model temperature and these preprocessing techniques on accuracy and hallucination rates.
Results: Spatial vs. Temporal Reasoning
The study revealed a significant performance gap between spatial and temporal reasoning, a phenomenon the authors describe as "unbalanced hallucinations." While models achieved a modest 22% accuracy on spatial queries (identifying locations), they struggled immensely with temporal logic, plummeting to just 5.5% accuracy. Models frequently failed to sequence events correctly or calculate the duration of disruptions.
Discussion: RAG and Multilingualism
The research highlighted several critical paths for improving industrial LLM applications:
- English Reasoning Bias: Translating context into English significantly improved spatial reasoning (p=0.0021), suggesting that LLMs reason more robustly in their dominant training language even when they are technically multilingual.
- RAG as a Factual Anchor: Retrieval-Augmented Generation (RAG) proved highly effective at reducing "out-of-bounds" hallucinations—such as inventing non-existent subway stations. By anchoring the model in the provided dataset, RAG shifts the distribution of responses toward factual correctness, though it does not inherently improve the models' underlying logical reasoning for complex tasks like pathfinding.
- Temperature Sensitivity: For structured industrial tasks, lower temperature settings are essential. Higher randomness directly correlated with increased hallucination rates and decreased factual reliability.
Conclusion and Future Work
This research underscores the challenges of deploying LLMs in industrial spatio-temporal contexts. While tools like RAG and strategic preprocessing can mitigate factual hallucinations, the "temporal trap" remains a significant hurdle for current architectures. Future work will focus on expanding the benchmark to other global cities and refining the logical reasoning capabilities of models in high-stakes environments to ensure safety and reliability.
Summary updated