Generative AI for Geospatial Analysis: Fine-Tuning ChatGPT to Convert Natural Language into Python-Based Geospatial Computations

Zachary Sherman, Sandesh Sharma Dulal, Jin-Hee Cho, Mengxi Zhang, Junghwan KimISPRS Int. J. Geo Inf 2025 · DOI

TL;DR

This paper introduces a fine-tuned version of OpenAI's GPT-40-mini model that significantly improves its ability to translate natural language queries into executable Python code for geospatial analysis. By addressing key limitations of off-the-shelf LLMs, this approach offers a more accurate, efficient, and user-friendly way to interact with complex spatial data, paving the way for enhanced smart city management and geospatial science applications.

What Problem Does This Paper Address?

Traditional Geographic Information System (GIS) workflows, while powerful, often present a steep learning curve for non-technical users like policymakers or urban planners. This lack of usability and scalability creates a barrier to entry for leveraging spatial data in decision-making. While large language models (LLMs) like ChatGPT offer a promising avenue for natural language interaction with data, existing LLMs often struggle with the nuances of geospatial reasoning. They can produce inaccurate results, "hallucinate" data, or generate code that fails to execute due to a lack of domain-specific understanding. This paper aims to bridge this gap by fine-tuning an LLM to specifically handle geospatial queries, making spatial analysis more accessible and reliable.

Methods

The core of this research lies in fine-tuning OpenAI's GPT-40-mini model. The authors developed a novel framework that goes beyond simply feeding the model more data. Key methodological innovations include:

  • Domain-Specific Fine-Tuning: The model was trained on a curated dataset of over 600 prompt-response pairs specifically designed for geospatial Python scripting. This dataset included realistic spatial operations like spatial joins, buffering, and nearest-neighbor queries.
  • Conversational Adaptability: The fine-tuned model can distinguish between general inquiries and computational queries, providing text-based responses for the former and executable Python code for the latter.
  • Location Identification and Correction: A fuzzy matching mechanism (using the Fuzzywuzzy package) was implemented to correct misspelled or improperly formatted geographic names, significantly reducing errors caused by input variations.
  • External Function Calls: Instead of generating monolithic scripts, the fine-tuned model calls predefined functions for common geospatial tasks (e.g., geocoding, isochrone generation). This modular approach drastically reduces token usage and improves computational efficiency.
  • Structured Output Protocols: Instructions were given to ensure generated code was directly executable, syntactically valid, and consistently formatted, minimizing execution errors.

The authors evaluated both an "As-Is" (unmodified) GPT-40-mini model and their fine-tuned version.

Datasets

The study utilized the following datasets for training and evaluation:

  • Virginia Geospatial Data:
    • Polygon shapefiles of county and ZIP code boundaries (U.S. Census Bureau TIGER/Line database).
  • A randomly sampled dataset of street addresses in Virginia.
  • A point shapefile of hospital locations in Virginia (Virginia Geographic Information Network - VGIN).
  • New York Geospatial Data: Used for external validation to assess the model's geographic generalizability.

The authors created a benchmark of six predefined geospatial queries designed to test various spatial analysis tasks, which were executed 100 times each for both models.

Key Insights and Ideas

  • Fine-tuning is Crucial for Geospatial LLMs: The stark performance difference between the "As-Is" and fine-tuned models highlights that general-purpose LLMs are insufficient for reliable geospatial analysis without specialized training.
  • Beyond Code Generation: Spatial Reasoning: The fine-tuned model doesn't just generate code; it demonstrates improved spatial reasoning, enabling it to handle more complex multi-step queries that the baseline model struggled with.
  • Efficiency Gains are Significant: The reduction in token usage is not just a cost-saving measure; it indicates a more efficient and scalable approach to processing complex spatial data.
  • User-Friendly GIS is Achievable: By translating natural language into executable code, LLMs can democratize access to powerful GIS tools, empowering non-experts to perform sophisticated spatial analyses.

Key Results

The fine-tuned GPT-40-mini model demonstrated remarkable improvements:

  • Accuracy: Achieved an overall accuracy of 89.7%, a 49.2 percentage point improvement over the "As-Is" model's 40.5%. For complex queries like driving-time accessibility, accuracy increased by over 6 times.
  • Execution Success: Significantly reduced execution failures, resolving issues like spatial joins entirely where the baseline model failed.
  • Token Efficiency: Reduced average token usage by approximately 74% compared to the "As-Is" model, leading to substantial cost savings and improved processing speed.
  • Error Reduction: Substantially decreased syntax errors, incorrect variable references, and CRS issues.

External validation with New York data confirmed the model's generalizability, showing similar high accuracy and efficiency gains.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Geographic Diversity: The model was primarily trained on Virginia data. Expanding the training dataset to include more diverse geographic areas and boundary hierarchies could further improve robustness.
  • Query Complexity: While the study covered foundational spatial queries, future work could explore more complex operations like spatiotemporal analysis, multi-layer joins, and raster-based analyses.
  • Model Benchmarking: The study focused on OpenAI models. Broader benchmarking across other LLMs, including open-source alternatives, is needed.
  • Conversational Memory: The current framework lacks conversational memory, limiting its ability to handle iterative or chained queries within a single session. Integrating this would be beneficial for dashboard applications.
  • API Costs: Reliance on commercial APIs (Mapbox, ArcGIS) incurs costs. Exploring open-source alternatives could enhance cost-efficiency.
  • Semantic Accuracy and Variability: While execution accuracy improved, the study acknowledges the need for further work on detecting semantic errors and assessing output variability for semantically equivalent queries. Future research should also focus on trust and uncertainty quantification.

Summary updated

BibTeX
@article{sherman2025generative,
  author = {Zachary Sherman and Sandesh Sharma Dulal and Jin-Hee Cho and Mengxi Zhang and Junghwan Kim},
  title = {{Generative AI for Geospatial Analysis: Fine-Tuning ChatGPT to Convert Natural Language into Python-Based Geospatial Computations}},
  journal = {ISPRS Int. J. Geo Inf.},
  year = {2025},
  doi = {10.3390/ijgi14080314},
}
Figure