Mr4Soil: A MapReduce-Based Framework Integrated with GIS for Soil Erosion Modelling

Zhigang Han, Fen Qin, Caihui Cui, Yannan Liu, Lingling Wang, Pinde FuISPRS Int. J. Geo Inf 2019 · DOI

TL;DR

This paper introduces Mr4Soil, a novel MapReduce-based framework designed to tackle the computational challenges of high-resolution soil erosion modeling. By leveraging parallel computing and integrating with GIS, Mr4Soil significantly enhances efficiency and scalability for analyzing large geospatial datasets, making refined soil erosion modeling more accessible.

What Problem Does This Paper Address?

Soil erosion modeling is crucial for understanding and mitigating environmental impacts, but it's hampered by the increasing volume and resolution of geospatial data (like DEMs, land use, and soil types). Traditional serial computing methods struggle to process these massive datasets efficiently, limiting the accuracy and scale of soil erosion predictions. This gap prevents the widespread application of refined models that could better inform conservation efforts and land management.

Methods

The Mr4Soil framework is built on the Hadoop MapReduce platform and comprises three layers:

  • Methodology Layer: Defines two parallel data-splitting strategies:
  • Row-oriented splitting: Divides raster data into horizontal segments, suitable for local and focal operations where computations for a cell are independent or depend on immediate neighbors.
  • Sub-basin-oriented splitting: Divides the basin into hydrological sub-basins, ideal for zonal operations where computations depend on larger, irregular areas.
  • Algorithm Layer: Implements six parallel computing operators for key geoprocessing tasks: spatial interpolation, vector rasterization, map algebra, slope gradient, flow direction, and slope length. These operators are designed to work with the MapReduce paradigm.
  • Application Layer: Integrates the parallel computing framework with a Geographic Information System (GIS) platform (ArcGIS) through a geoprocessing toolbox. This allows users to initiate parallel computations and visualize results within a familiar GIS environment, using Python scripts and the ArcPy library.

The framework converts input data (vector and raster) into GeoCSV or key-value pair formats for efficient processing by Hadoop.

Datasets

The study utilizes data for the Gushanchuan basin in the Loess Plateau, China. This includes:

  • Raster Data:
    • Digital Elevation Model (DEM) at various spatial resolutions (30m, 10m, 5m, 2.5m).
  • Land-use type data.
  • Normalized Difference Vegetation Index (NDVI) data.
  • Vector Data:
    • Soil types (polygon).
  • Soil and water conservation engineering measures (polygon).
  • Rainfall data (point) from rain gauge stations.
  • Sub-basin boundary data (polygon).

The paper does not introduce new benchmark datasets but uses existing data to demonstrate the framework's capabilities.

Key Insights and Ideas

  • Scalability is Key: As data resolution and volume increase, the parallel processing approach of Mr4Soil shows a significant improvement in computational efficiency, far surpassing serial methods.
  • Tailored Splitting Strategies: The choice between row-oriented and sub-basin-oriented data splitting is crucial for optimizing performance based on the type of computational task (local, focal, or zonal).
  • GIS Integration is Essential: Bridging the gap between powerful parallel computing platforms like Hadoop and user-friendly GIS environments is vital for practical application in fields like soil erosion modeling. The geoprocessing toolbox makes complex parallel computations accessible to GIS users.
  • Hadoop's Advantage for Big Data: For large-scale, high-resolution geospatial analysis, Hadoop's distributed computing model offers a cost-effective and scalable solution compared to specialized hardware like GPUs, especially as data volume grows.

Key Results

  • The Mr4Soil framework demonstrated significant performance improvements with increasing data resolution and cluster size.
  • For 2.5m resolution data, the maximum parallel acceleration ratio reached 12.52 (for the K factor).
  • As the number of worker nodes increased (from 4 to 16), calculation times generally decreased, with notable reductions for complex tasks like LS factor calculation and overall model computation (model A).
  • For 2.5m resolution data, the time consumption for model A decreased by 1.50 and 1.34 minutes when increasing nodes from 4 to 8 and 16, respectively.
  • While GPU and CPU were faster for very small datasets, Hadoop's performance scaled much better with increasing data volume, making it superior for large-scale analyses.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that for very low resolutions (30m), some factors showed acceleration ratios less than 1.0, attributed to communication overhead during data splitting and merging.
  • The current implementation focuses on a desktop GIS platform and a spatially distributed model.
  • Future Work:
    • Develop a service interface for soil erosion models with parallel computing tasks, potentially based on web services for broader accessibility.
  • Extend the framework to support physically based soil erosion models, further enhancing its applicability.
  • Explore further optimization of communication overhead in data splitting and merging for even greater efficiency.

Summary updated

BibTeX
@article{han2019mr,
  author = {Zhigang Han and Fen Qin and Caihui Cui and Yannan Liu and Lingling Wang and Pinde Fu},
  title = {{Mr4Soil: A MapReduce-Based Framework Integrated with GIS for Soil Erosion Modelling}},
  journal = {ISPRS Int. J. Geo Inf.},
  year = {2019},
  doi = {10.3390/ijgi8030103},
}
Figure