Geo-Expert: Towards Expert-Level Geological Reasoning via Parameter-Efficient Fine-Tuning

Chenyou Guo, Zongqi Liu, Yizhou Zhang, Zhaorui Jiang, Ze LiuAI4Science @ ICML 2026 · OpenReview

TL;DR

The Geo-Expert project introduces a family of parameter-efficient Large Language Models (LLMs) specifically designed for complex geological reasoning. By fine-tuning smaller models (8B to 32B) on a high-quality, textbook-derived instruction dataset, the researchers created a domain-specific model (Geo-Expert 8B) that outperforms general-purpose giants like Llama-3.1-70B and even proprietary models like GPT-4o in specialized geological tasks.

What Problem Does This Paper Address?

While general-purpose LLMs are impressive, they often "hallucinate" when faced with the specialized logic of solid-earth geology. Most current AI applications in Earth sciences focus on surface-level data like remote sensing or GIS. However, understanding the "solid Earth"—interpreting subsurface stratigraphic sequences, reconstructing tectonic evolution over millions of years, or analyzing rock mechanics—requires deep-domain reasoning that general models lack.

A major hurdle is polysemy: words that have common meanings in everyday life but very specific meanings in geology. For example, a general model might interpret a "wedge" as a simple tool or a mechanical component, whereas a geologist sees it as a specific structural feature that dictates stress distribution and faulting.

Methods

The authors developed a reproducible pipeline to transform static scientific knowledge into "reasoning-aware" training data:

  1. Instruction Synthesis Pipeline: They used a "document-to-instruction" workflow. This involves Chapter-Aware Recursive Chunking (splitting textbooks by semantic headers rather than arbitrary lengths) and Domain-Structured Question Generation to ensure broad coverage of geological concepts.
  2. Chain-of-Thought (CoT) Distillation: To teach the models how to reason, they used a "teacher" model (DeepSeek-R1) to generate intermediate reasoning traces. These traces show the step-by-step logic needed to reach a geological conclusion, which is then used to fine-tune the "student" models.
  3. Parameter-Efficient Fine-Tuning (PEFT): Instead of retraining entire models, they used LoRA (Low-Rank Adaptation). This allowed them to adapt models like Qwen3 and Gemma-3 by only updating a small fraction of the parameters, making it possible to train 27B+ parameter models on consumer-grade hardware (like a 4×RTX 5090 setup).

Datasets

  • Training Set: A custom-curated dataset of 11,518 high-quality instruction pairs derived from five canonical geology textbooks covering sequence stratigraphy, structural geology, tectonics, and mapping.
  • Geo-Eval Benchmark: A new, novel benchmark consisting of 387 expert-vetted "hard boundary" questions. These questions were specifically chosen through an adversarial mining process to identify areas where general-purpose models typically fail but domain-aligned models succeed.

Key Insights and Ideas

  • Domain Alignment > Parameter Count: The most striking finding is that a domain-aligned 8B model can outperform a 70B generalist. This suggests that for "vertical" (specialized) fields, high-quality, targeted data is more impactful than sheer model size.
  • The "Wedge" Test: Qualitative analysis showed that GPT-4o failed a question about "wedge thickening" by suggesting carbon fiber reinforcement (a civil engineering solution). In contrast, Geo-Expert correctly identified geological mechanisms like thrust fault sliding and lateral material flow.
  • Reasoning vs. Retrieval: The paper emphasizes that scientific adaptation must move beyond "factual recall" (memorizing definitions) toward "structured reasoning" (chaining principles to solve a problem).

Key Results

  • Performance Breakthrough: Fine-tuning led to massive gains. For instance, the Qwen3-32B-geo model improved its average score from 5.00 to 6.82 on the Geo-Eval benchmark.
  • Beating the Giants: The Qwen3-8B-geo model achieved a score of 6.27, surpassing the proprietary GPT-4o (5.93) and the open-weight Llama-3.1-70B (4.12).
  • Efficiency: The 8B model demonstrated the highest "score-per-billion-parameter" ratio, making it a highly practical candidate for local deployment in research or industrial settings where data privacy or compute costs are concerns.

Limitations and Future Work

  • Text-Only Modality: Geology is inherently visual (cross-sections, seismic logs, field photos). The current Geo-Expert is text-only, and the authors aim to integrate Vision-Language Models (VLMs) in the future.
  • Subdiscipline Bias: The training data is currently weighted toward structural geology and tectonics; fields like geochemistry and geophysics are currently under-represented.
  • Static Knowledge: The model is a "static reasoning engine." Future iterations may incorporate Retrieval-Augmented Generation (RAG) to allow the model to stay updated with the latest geological literature.

Summary updated

Abstract

While general-purpose Large Language Models (LLMs) applied to Geology often hallucinate when reasoning about subsurface structures and deep-time evolution, current AI in Earth sciences predominantly targets surface remote sensing and GIS. To bridge this gap, we introduce Geo-Expert, a family of parameter-efficient geological LLMs fine-tuned on a custom-curated, high-quality instruction dataset processed with a reproducible synthesis workflow and geology-specific prompts and filters. We investigate the impact of model scaling and architecture by fine-tuning three base models: Qwen3-8B, Qwen3-32B, and Gemma-3-27B, with Low-Rank Adaptation (LoRA) method. Our extensive evaluation on a novel domain-specific benchmark, Geo-Eval, reveals that a domain-aligned 8B model can outperform open-weight 70B generalists and proprietary GPT-4o on specialized geological reasoning, while a 32B variant approaches frontier reasoning models. The optimized 8B model further offers a favorable parameter-efficiency profile for deployment. This work provides a reproducible recipe for democratizing scientific LLMs and establishes a baseline for geological artificial intelligence.
BibTeX
@inproceedings{guo2026geo,
  author = {Chenyou Guo and Zongqi Liu and Yizhou Zhang and Zhaorui Jiang and Ze Liu},
  title = {{Geo-Expert: Towards Expert-Level Geological Reasoning via Parameter-Efficient Fine-Tuning}},
  booktitle = {AI4Science @ ICML},
  year = {2026},
  url = {https://openreview.net/forum?id=rwC2iKytOa},
}
Figure