TL;DR
The Matsu Wheel is a novel cloud-based framework designed for the efficient analysis of massive Earth satellite imagery datasets. By processing data in a "scanning query" fashion, it avoids redundant data reads, enabling faster and more resource-efficient analysis for applications like disaster detection and relief.
What Problem Does This Paper Address?
The paper tackles the growing challenge of analyzing the ever-increasing volume of scientific data, particularly from Earth observation satellites. Traditional query methods are inefficient for tasks that require processing an entire dataset (e.g., analyzing every image for a specific phenomenon). This bottleneck hinders scientific discovery and the timely application of data for critical purposes like disaster response. The authors aim to provide a solution that can efficiently process and reanalyze large collections of hyperspectral satellite imagery.
Methods
The core innovation is the "Matsu Wheel" framework, which is built on an open-source cloud infrastructure using OpenStack and Hadoop. The key idea is to process data in a "scanning query" manner, where the entire dataset is read and processed once, and multiple analytics are applied sequentially to this data chunk. This contrasts with traditional approaches that might re-read the data for each individual analysis.
The framework involves:
- Cloud Environment: Utilizing OpenStack for computing and Hadoop for distributed storage (HDFS) and processing.
- Data Pre-processing: Data from NASA's EO-1 satellite is ingested, converted to a MapReduce-friendly format (SequenceFile), and stored in HDFS. Metadata is extracted and stored in Accumulo (a NoSQL database) for geospatial indexing.
- The "Wheel" Architecture: Analytics are designed to "plug in" to the Wheel. As new data arrives, it's processed by a series of analytics in a pipeline. Intermediate results are stored, and secondary analyses can be performed on these results.
- Analytics: The paper details several specific analytics developed for the Wheel:
- Contours and Clusters: Identifies spectrally anomalous regions in hyperspectral data.
- Rare Pixel Finder (RPF): Detects small clusters of unusual pixels.
- Gaussian Mixture Model and K-Nearest Neighbors (GMM-KNN): Another method for finding small clusters of unusual pixels.
- Spectral Blobs: Uses a windowing technique to identify spectrally similar regions.
- Supervised Spectral Classifier: A land cover classification algorithm, particularly useful for water and flood detection.
Datasets
The primary dataset used is daily hyperspectral satellite imagery from NASA's Earth Observing-1 (EO-1) satellite, specifically data from the Advanced Land Imager (ALI) and Hyperion instruments. The paper mentions using Level 1G data in GeoTIFF format. For the Supervised Spectral Classifier, training data was constructed from sections of EO-1 Hyperion images with known land coverage and cloud coverage.
Key Insights and Ideas
- The "Scan Once, Analyze Many" Paradigm: The most significant insight is the efficiency gained by processing the entire dataset once for multiple scanning queries, rather than re-reading the data for each query. This is crucial for large-scale satellite imagery analysis.
- Modularity and Extensibility: The "Wheel" architecture is designed to be highly modular, allowing new analytics to be easily integrated without modifying the core framework. This promotes a collaborative development environment.
- Bridging Data and Decision-Makers: The framework not only processes raw data but also generates accessible visual reports and OGC-compliant Web Map Services, making the insights available to the public and decision-makers for applications like disaster relief.
- Cloud-Native for Big Data: The reliance on OpenStack and Hadoop demonstrates a practical approach to leveraging cloud computing for handling the massive scale of Earth observation data.
Key Results
The paper describes the successful implementation of the Matsu Wheel for processing daily EO-1 data. The developed analytics have been used to identify interesting phenomena, such as volcanic events (e.g., Barren Island volcano). The Supervised Spectral Classifier showed promising results in distinguishing land cover types, with visual comparisons demonstrating consistency with existing methods, though some discrepancies in cloud coverage calculations were noted. The framework is designed to handle large datasets, with Hyperion scenes alone generating 1.5-2.5 GB of data per scene.
Limitations and Future Work
- Limitations: The paper acknowledges that the Supervised Spectral Classifier's cloud coverage calculations may not perfectly align with USGS estimates, potentially due to differing treatments of thin cloud "haze." The performance of distributed systems can also be subject to variability due to shared workloads across virtual machines.
- Future Work: The authors plan to extend the Supervised Spectral Classifier for specific applications like floodplain analysis for flood detection and developing fire detection analytics. They also aim to move beyond detection to prediction, proposing a framework for flood prediction that involves analyzing topological data, identifying flood basins, and correlating water coverage with water volume and rainfall. The goal is to continue developing the Matsu Wheel analytics and apply the framework to other Earth satellite datasets.
Summary updated