TL;DR
This paper tackles the challenge of storing and accessing massive amounts of remote sensing data, which traditional relational databases struggle with. They propose a distributed storage and access approach using MongoDB, separating structured metadata into documents and unstructured image data into GridFS, demonstrating improved storage and access performance.
What Problem Does This Paper Address?
The exponential growth of earth-observation data, driven by advancements in remote sensing and satellite technology, has created a "big data era" in this field. Traditional relational databases, with their rigid schemas and limitations in handling large volumes of unstructured data, are proving inadequate for efficiently storing, managing, and accessing this ever-increasing data deluge. This poses a significant bottleneck for applications in environmental monitoring, land resource surveys, and disaster assessment. The paper aims to bridge this gap by leveraging the flexibility and scalability of NoSQL databases, specifically MongoDB, to create a more efficient system for handling massive remote sensing datasets.
Methods
The core of the proposed solution lies in a distributed storage and access architecture built upon MongoDB. Key methodological components include:
- Distributed Cluster Architecture: A distributed cluster is established using MongoDB's sharding technology. This allows data to be partitioned and distributed across multiple machines, enhancing scalability and load balancing. The architecture incorporates replica sets for high availability and consistency.
- Archiving Model for Metadata: To address the heterogeneity of remote sensing metadata, a unified archiving model is constructed. This model standardizes the structure of metadata, drawing inspiration from international standards like ISO 19115 and CSDGM.
- Separated Storage of Metadata and Image Data:
- Metadata: Structured metadata is stored as documents within MongoDB collections.
- Image Data: Unstructured image data is managed using MongoDB's GridFS mechanism. GridFS is a lightweight file storage specification that splits large binary files into smaller chunks, storing them as documents. This approach is particularly useful for handling large files that exceed MongoDB's default document size limit.
- Shard Key Strategy: The paper investigates the impact of different shard key strategies on performance. Specifically, it compares using a hashed
_id(for random distribution) with using a spatially relevant field likeTopLeftLatitudefor metadata storage. For image data, a combined shard key offiles_idandnis used.
Datasets
The experimental data consists of:
- Remote Sensing Metadata: Approximately 100 million global metadata records from 2013 to 2017, obtained from the USGS website.
- Remote Sensing Image Data: Various satellite data including Landsat, FY, and Sentinel data. Specific examples include Landsat 5 and 8, FY3A, FY3B, FY3C, Sentinel 1, Sentinel 2, and Sentinel 3.
Key Insights and Ideas
- NoSQL for Big Geospatial Data: The paper strongly advocates for NoSQL databases like MongoDB as a superior solution for managing massive geospatial datasets compared to traditional relational databases.
- GridFS for Large Binary Files: MongoDB's GridFS is a highly effective mechanism for storing and managing large binary files like remote sensing images, overcoming the limitations of standard document storage.
- Metadata Standardization is Crucial: The heterogeneity of metadata across different satellite sources necessitates a standardized archiving model for unified management.
- Shard Key Choice Matters: The selection of a shard key significantly impacts both storage and access performance. While a spatially relevant key like
TopLeftLatitudecan be intuitive, a hashed key often provides better load balancing and overall performance for random access. - Separation of Concerns: Storing metadata and image data separately, with appropriate mechanisms for each, is key to optimizing the system.
Key Results
- Metadata Storage: The
_id_hashedshard key strategy generally yielded better storage performance (higher insert speed) than usingTopLeftLatitude, especially as the volume of metadata increased. - Metadata Access: The
_id_hashedshard key strategy demonstrated significantly better access performance (lower access time) compared toTopLeftLatitude, particularly for large query ranges. For 1,000,000 metadata records,_id_hashedwas 48 times faster. - Image Data Storage: MongoDB outperformed PostgreSQL in storing large remote sensing image data, with PostgreSQL taking up to 1.6 times longer for the same data. MongoDB's storage performance was also more stable.
- Image Data Access: MongoDB also showed superior performance in accessing large remote sensing image data compared to PostgreSQL, with PostgreSQL taking up to 2.8 times longer.
- Comparison with Prior Work: The proposed "Method 1" (using
files_idas a shard key for image data) showed improved access performance over "Method 2" (usingnas a shard key), with data access time decreasing by up to 69.4%.
Limitations and Future Work
- Shard Key Optimization: The paper acknowledges that neither of the tested shard key strategies guarantees optimal performance for both storage and access simultaneously. Future work should consider practicality and explore more advanced shard key strategies.
- Network Bandwidth Influence: The performance of the MongoDB cluster is influenced by network bandwidth, which was not extensively studied.
- Number of Data Nodes: The authors plan to investigate the influence of the number of data nodes on the performance of the distributed system in future work.
Summary updated