Supporting Land Reuse of Former Open Pit Mining Sites using Text Classification and Active Learning

Christopher Schröder, Kim Bürgl, Yves Annanias, Andreas Niekler, Lydia Müller, Daniel Wiegreffe, Christian Bender, Christoph Mengs, Gerik Scheuermann, Gerhard HeyerACL 2021

TL;DR

This paper tackles the challenge of efficiently extracting crucial information from vast archives of unstructured text documents related to former open-pit mining sites. By combining OCR, active learning for multi-label text classification, and GIS visualization, they enable faster and more informed land reuse decisions, contributing to sustainable development goals.

What Problem Does This Paper Address?

Former open-pit mining sites often leave behind hazardous or contaminated land, requiring extensive documentation to ensure safe and effective reuse. In regions like eastern Germany, this information is buried in thousands of legacy documents, many of which are unstructured or even paper-based. Manually sifting through these reports to identify restrictions, hazards, and other critical details is a time-consuming and laborious process, significantly delaying land reuse efforts. This delay not only hinders economic development but also impedes progress towards several United Nations Sustainable Development Goals (SDGs) related to clean water, sustainable cities, and life on land. The paper aims to automate this information extraction process to accelerate land reuse.

Methods

The proposed workflow integrates three key components:

  • Optical Character Recognition (OCR): For non-digitized documents, Tesseract is used to extract text. The authors acknowledge the challenges of noisy OCR output due to document variability and focus on pre-processing steps like orientation analysis, denoising, and binarization to improve confidence scores.
  • Active Learning for Multi-Label Text Classification: Since no pre-labeled data was available, active learning is employed. This iterative process involves a human annotator labeling a small subset of sentences, training a model, and then having the model select the most informative unlabeled sentences for further annotation. This approach aims to minimize the overall labeling effort. The authors use a transformer-based model (gbert-base) for classification, trained on multi-hot encoded vectors for both restriction and topic labels.
  • Geographic Information System (GIS) Visualization: The extracted and classified information is integrated with existing geospatial data. This allows for the visualization of restrictions and topics directly on maps, enabling experts to query and analyze information relevant to specific areas.

Datasets

The primary dataset consists of 31,605 documents related to former lignite open-pit mines in eastern Germany, provided by the Lausitzer und Mitteldeutsche Bergbauverwaltungsgesellschaft mbH (LMBV). This dataset includes over 30,000 geographic features. For training and evaluation, a subset of 2,000 sentences was manually annotated by three non-expert annotators, with inter-annotator agreement measured using Krippendorff's alpha. The annotated dataset was split into training (500), validation (500), and test (1000) sets using iterative stratification. An unlabeled pool of 312,299 sentences was used for the active learning process.

Key Insights and Ideas

  • Active learning is crucial for low-resource domains: The absence of pre-labeled data for this specific domain made active learning the most viable approach to build a classification model efficiently.
  • Co-occurrence of labels is complex and informative: The active learning process revealed a richer set of label co-occurrences than initially captured by keyword-based bootstrapping. This suggests that understanding how different restrictions and topics appear together is vital for a comprehensive analysis.
  • GIS integration transforms data into actionable insights: Simply extracting text is not enough; visualizing this information within a GIS context allows experts to directly answer queries about specific areas and make informed decisions about land reuse.
  • Human expertise remains critical, but can be augmented: While the system automates much of the process, the need for human annotators in active learning and the acknowledgment of potential misclassifications highlight the continued importance of human oversight and domain expertise.

Key Results

  • Restrictions: The active learning approach achieved high F1 scores for restriction categories, with an average micro-F1 of 0.87 and macro-F1 of 0.90. Labels like "Prohibition" and "Restricted Area" performed particularly well.
  • Topics: Topic-oriented categories showed more moderate performance, with an average micro-F1 of 0.71 and macro-F1 of 0.71. Labels like "Weather" and "Construction" saw significant improvements due to active learning, increasing by 5-21 percentage points.
  • OCR Improvement: Pre-processing steps improved the OCR confidence score to 97% of pages exceeding a 75% threshold for the Lausitz region, a significant jump from 45% without pre-processing.

Limitations and Future Work

  • Recall for False Negatives: The authors acknowledge that achieving 100% recall to minimize false negatives remains a challenge.
  • Label Sparsity: The inherent sparsity of some labels makes creating a truly representative test set difficult and would require massive annotation efforts.
  • Misclassification Impact: The potential for misclassifications to lead to incorrect decisions or endanger individuals is a significant ethical consideration. The authors propose quality assurance processes to mitigate this.
  • Job Displacement: The automation of document review could theoretically lead to a decline in demand for human readers, though the authors argue that specialists are not easily replaceable.
  • Future Work: The paper suggests further research into improving recall to minimize false negatives and correcting false positives. Exploring more advanced techniques for handling label sparsity and further refining the GIS visualization and interaction capabilities are also promising directions.

Summary updated

BibTeX
@inproceedings{schroder2021supporting,
  author = {Christopher Schröder and Kim Bürgl and Yves Annanias and Andreas Niekler and Lydia Müller and Daniel Wiegreffe and Christian Bender and Christoph Mengs and Gerik Scheuermann and Gerhard Heyer},
  title = {{Supporting Land Reuse of Former Open Pit Mining Sites using Text Classification and Active Learning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure