A Distributed Storage and Access Approach for Massive Remote Sensing Data in MongoDB

Shuang Wang, Guoqing Li, Xiaochuang Yao, Yi Zeng, Lushen Pang, Lianchong ZhangISPRS Int. J. Geo Inf 2019 · DOI

TL;DR

This paper tackles the challenge of storing and accessing massive amounts of remote sensing data, which traditional relational databases struggle with. They propose a distributed storage and access approach using MongoDB, separating structured metadata into documents and unstructured image data into GridFS, demonstrating improved storage and access performance.

What Problem Does This Paper Address?

The exponential growth of earth-observation data, driven by advancements in remote sensing and satellite technology, has created a "big data era" in this field. Traditional relational databases, with their rigid schemas and limitations in handling large volumes of unstructured data, are proving inadequate for efficiently storing, managing, and accessing this ever-increasing data deluge. This poses a significant bottleneck for applications in environmental monitoring, land resource surveys, and disaster assessment. The paper aims to bridge this gap by leveraging the flexibility and scalability of NoSQL databases, specifically MongoDB, to create a more efficient system for handling massive remote sensing datasets.

Methods

The core of the proposed solution lies in a distributed storage and access architecture built upon MongoDB. Key methodological components include:

  • Distributed Cluster Architecture: A distributed cluster is established using MongoDB's sharding technology. This allows data to be partitioned and distributed across multiple machines, enhancing scalability and load balancing. The architecture incorporates replica sets for high availability and consistency.
  • Archiving Model for Metadata: To address the heterogeneity of remote sensing metadata, a unified archiving model is constructed. This model standardizes the structure of metadata, drawing inspiration from international standards like ISO 19115 and CSDGM.
  • Separated Storage of Metadata and Image Data:
    • Metadata: Structured metadata is stored as documents within MongoDB collections.
  • Image Data: Unstructured image data is managed using MongoDB's GridFS mechanism. GridFS is a lightweight file storage specification that splits large binary files into smaller chunks, storing them as documents. This approach is particularly useful for handling large files that exceed MongoDB's default document size limit.
  • Shard Key Strategy: The paper investigates the impact of different shard key strategies on performance. Specifically, it compares using a hashed _id (for random distribution) with using a spatially relevant field like TopLeftLatitude for metadata storage. For image data, a combined shard key of files_id and n is used.

Datasets

The experimental data consists of:

  • Remote Sensing Metadata: Approximately 100 million global metadata records from 2013 to 2017, obtained from the USGS website.
  • Remote Sensing Image Data: Various satellite data including Landsat, FY, and Sentinel data. Specific examples include Landsat 5 and 8, FY3A, FY3B, FY3C, Sentinel 1, Sentinel 2, and Sentinel 3.

Key Insights and Ideas

  • NoSQL for Big Geospatial Data: The paper strongly advocates for NoSQL databases like MongoDB as a superior solution for managing massive geospatial datasets compared to traditional relational databases.
  • GridFS for Large Binary Files: MongoDB's GridFS is a highly effective mechanism for storing and managing large binary files like remote sensing images, overcoming the limitations of standard document storage.
  • Metadata Standardization is Crucial: The heterogeneity of metadata across different satellite sources necessitates a standardized archiving model for unified management.
  • Shard Key Choice Matters: The selection of a shard key significantly impacts both storage and access performance. While a spatially relevant key like TopLeftLatitude can be intuitive, a hashed key often provides better load balancing and overall performance for random access.
  • Separation of Concerns: Storing metadata and image data separately, with appropriate mechanisms for each, is key to optimizing the system.

Key Results

  • Metadata Storage: The _id_hashed shard key strategy generally yielded better storage performance (higher insert speed) than using TopLeftLatitude, especially as the volume of metadata increased.
  • Metadata Access: The _id_hashed shard key strategy demonstrated significantly better access performance (lower access time) compared to TopLeftLatitude, particularly for large query ranges. For 1,000,000 metadata records, _id_hashed was 48 times faster.
  • Image Data Storage: MongoDB outperformed PostgreSQL in storing large remote sensing image data, with PostgreSQL taking up to 1.6 times longer for the same data. MongoDB's storage performance was also more stable.
  • Image Data Access: MongoDB also showed superior performance in accessing large remote sensing image data compared to PostgreSQL, with PostgreSQL taking up to 2.8 times longer.
  • Comparison with Prior Work: The proposed "Method 1" (using files_id as a shard key for image data) showed improved access performance over "Method 2" (using n as a shard key), with data access time decreasing by up to 69.4%.

Limitations and Future Work

  • Shard Key Optimization: The paper acknowledges that neither of the tested shard key strategies guarantees optimal performance for both storage and access simultaneously. Future work should consider practicality and explore more advanced shard key strategies.
  • Network Bandwidth Influence: The performance of the MongoDB cluster is influenced by network bandwidth, which was not extensively studied.
  • Number of Data Nodes: The authors plan to investigate the influence of the number of data nodes on the performance of the distributed system in future work.

Summary updated

BibTeX
@article{wang2019distributed,
  author = {Shuang Wang and Guoqing Li and Xiaochuang Yao and Yi Zeng and Lushen Pang and Lianchong Zhang},
  title = {{A Distributed Storage and Access Approach for Massive Remote Sensing Data in MongoDB}},
  journal = {ISPRS Int. J. Geo Inf.},
  year = {2019},
  doi = {10.3390/ijgi8120533},
}
Figure