TL;DR
This paper introduces a multi-agent LLM system designed to enhance decision-making in smart city management. By integrating LLMs with existing urban data and services, the system achieves significantly higher accuracy and efficiency in processing complex urban planning queries compared to standalone LLMs, paving the way for more informed urban governance.
What Problem Does This Paper Address?
Modern cities generate vast amounts of complex, heterogeneous data from various sources like urban planning documents, geospatial information, and real-time service APIs. Extracting actionable insights from this data for strategic decision-making is a significant challenge. While Large Language Models (LLMs) show promise in processing unstructured text, they often struggle with domain-specific knowledge, real-time data, and can "hallucinate" information. This paper addresses the gap in effectively leveraging LLMs for complex urban management tasks by proposing a multi-agent system that combines LLM capabilities with structured access to urban data and services, aiming to improve both the accuracy and efficiency of decision support.
Methods
The core of the proposed solution is a multi-agent system architecture. Individual LLM agents are designed with mechanisms for instruction processing, action execution, and reflection. These agents interact through two primary means: "tool usage" (where an agent calls external functions or APIs) and "agent-to-agent communication" (peer-to-peer interaction).
The system is structured into three layers:
- Interface Layer: A chatbot interface for user interaction.
- Orchestration Layer: An "agent-conductor" that coordinates the system's operations.
- Processing Layer: Specialized secondary agents for:
- Tool Calling: Interacting with city services via APIs.
- RAG Operations: Performing Retrieval-Augmented Generation using a vector database of municipal documents.
- Answer Summarization: Synthesizing and formatting final responses.
- Validation: Ensuring the integrity of answers.
The system utilizes Retrieval-Augmented Generation (RAG) to ground LLM responses in factual information from document databases and integrates with city service APIs for real-time data.
Datasets
The paper uses a validation dataset of 150 question-answer pairs, manually prepared by urban studies experts and augmented using GPT-40. This dataset is available in an open repository. The system also leverages existing data sources such as OpenStreetMap, the Territorial Development Fund's open data, and national statistics service data for St. Petersburg.
Key Insights and Ideas
- Multi-agent systems are key: Standalone LLMs struggle with the complexity and data requirements of smart city management. A multi-agent approach, where specialized agents handle different tasks (data retrieval, RAG, tool calling), is crucial for success.
- Context is king (especially for numbers): LLMs without access to relevant, factual context perform poorly, especially when numerical precision is required. Providing context significantly boosts accuracy.
- RAG is a powerful tool: Retrieval-Augmented Generation, especially when combined with local knowledge and regulations, dramatically improves response quality and reduces the tendency for LLMs to "hallucinate."
- Middle-sized LLMs can be sufficient: The study suggests that middle-sized LLM models can achieve high accuracy, potentially outperforming larger models when integrated into a well-designed multi-agent system.
- Hybrid approach for optimal results: While LLM multi-agent systems offer remarkable speed and efficiency for data analysis, human expertise remains vital for strategic, long-term planning and decision-making.
Key Results
- Pipeline Selection Accuracy: The multi-agent system achieved pipeline selection accuracy of 94-99% across different LLM models.
- Response Accuracy Improvement:
- RAG integration improved response accuracy by 17% for strategic development queries and 55% for service accessibility questions.
- The combined use of document databases and service APIs resulted in the highest G-Eval scores of 0.68–0.74, significantly outperforming standalone LLM responses (0.30–0.38).
- Efficiency: The system can complete tasks in hours that would typically take human teams days or weeks.
- Relevance: The system maintained high relevance in its answers (scores of 0.91–0.95).
Limitations and Future Work
- Computational Scalability: Deploying LLMs requires substantial computational resources, posing a trade-off between cloud infrastructure (with security risks) and local deployment (with high hardware investment).
- Real-time Data Integration: Ensuring consistent, secure, and standardized data flow from diverse urban information systems remains a challenge.
- Agent System Stability: The reliability of the complex orchestration between agents, especially in production environments, is a concern, requiring careful prompt engineering and regular updates.
- Future Work:
- Enhancing agent coordination for complex urban planning workflows.
- Improving validation mechanisms for factual accuracy.
- Optimizing computational efficiency for resource-constrained municipalities.
- Developing specialized agents for emergency response and crisis management.
- Integrating real-time urban sensing networks.
- Improving temporal modeling capabilities for urban evolution tracking.
- Phased implementation for cities with varying digital maturity.
Summary updated