Introduction
The United States faces a significant environmental challenge with hundreds of thousands of "orphaned" oil and gas wells—abandoned sites with no known owner. These wells often leak methane and contaminate groundwater, necessitating urgent plugging efforts. However, identifying their precise locations is difficult because historical records are often trapped in unstructured formats, including typed PDFs, blurry 1950s-era scans, and documents obscured by stamps and handwriting. This paper presents an automated workflow leveraging Large Language Models (LLMs) to unlock this critical geoscientific data.
Methodology
The researchers developed a multi-stage information extraction pipeline designed to handle varying document qualities. The process begins with text extraction: clean PDFs are processed via pdftotext, while scanned and handwritten documents utilize Google’s Enterprise OCR (Document AI).
Following extraction, the text is fed into an LLM using specialized prompt engineering. The team tested several prompt levels, finding that "domain-aware" prompts—which include constraints like "longitude must be negative" for U.S. locations—significantly improved accuracy. The study primarily evaluated the Llama 2 family (7B, 13B, and 70B parameters) on local hardware, alongside benchmarks against Mixtral 8x7B and GPT-3.5. To improve coordinate conversion from Degrees-Minutes-Seconds to Decimal Degrees, a Chain-of-Thought (CoT) strategy was implemented, requiring the model to explain its reasoning steps.
Datasets and Evaluation
The workflow was tested on 160 historical records across two distinct categories:
- Colorado Dataset (150 records): Clean, digital, text-based PDF drilling completion reports.
- Pennsylvania Dataset (10 records): Highly challenging scanned images containing handwriting, overlapping stamps, and blurred text.
Models were evaluated using Exact Match accuracy ($A_{EM}$) and Offset accuracy ($A_{OS}$), the latter allowing for a 10-meter margin of error in location data.
Key Insights and Results
The study revealed that domain knowledge acts as a "force multiplier." Without specific rules, LLMs often struggled to distinguish between different types of depth (e.g., "measured depth" vs. "true vertical depth") or coordinate labels. Furthermore, standard OCR often disrupts the visual structure of tables, making it difficult for models to associate labels with their corresponding values.
Performance Highlights: * Digital Accuracy: Llama 2-70B achieved 100% accuracy on the clean Colorado records. * Scanned Document Performance: On the messy Pennsylvania records, Llama 2-70B reached 70% accuracy for location and 90% for depth. * Model Comparison: GPT-3.5 was the most robust, achieving 100% accuracy across all document types, largely due to its superior mathematical reasoning for coordinate transformations.
Conclusion and Future Work
While LLMs can replace months of human labor with hours of automated processing, limitations remain. The primary bottleneck is the quality of initial OCR; errors at this stage propagate through the LLM. Future research will explore multi-modal LLMs that can interpret document images directly, bypassing traditional OCR. Additionally, the team suggests using "function calling" to offload precise mathematical calculations to Python scripts, and fine-tuning smaller open-source models to achieve high-tier performance on more accessible hardware.
Summary updated