The Matsu Wheel: A Cloud-Based Framework for Efficient Analysis and Reanalysis of Earth Satellite Imagery

Maria T. Patterson, Nikolas Anderson, Collin Bennett, Jacob Bruggemann, Robert L. Grossman, Matthew Handy, Vuong Ly, Daniel J. Mandl, Shane Pederson, James Pivarski, Ray Powell, Jonathan Spring, Walt Wells, John Xia

TL;DR

The Matsu Wheel is a novel cloud-based framework designed for the efficient analysis of massive Earth satellite imagery datasets. By processing data in a "scanning query" fashion, it avoids redundant data reads, enabling faster and more resource-efficient analysis for applications like disaster detection and relief.

What Problem Does This Paper Address?

The paper tackles the growing challenge of analyzing the ever-increasing volume of scientific data, particularly from Earth observation satellites. Traditional query methods are inefficient for tasks that require processing an entire dataset (e.g., analyzing every image for a specific phenomenon). This bottleneck hinders scientific discovery and the timely application of data for critical purposes like disaster response. The authors aim to provide a solution that can efficiently process and reanalyze large collections of hyperspectral satellite imagery.

Methods

The core innovation is the "Matsu Wheel" framework, which is built on an open-source cloud infrastructure using OpenStack and Hadoop. The key idea is to process data in a "scanning query" manner, where the entire dataset is read and processed once, and multiple analytics are applied sequentially to this data chunk. This contrasts with traditional approaches that might re-read the data for each individual analysis.

The framework involves:

  • Cloud Environment: Utilizing OpenStack for computing and Hadoop for distributed storage (HDFS) and processing.
  • Data Pre-processing: Data from NASA's EO-1 satellite is ingested, converted to a MapReduce-friendly format (SequenceFile), and stored in HDFS. Metadata is extracted and stored in Accumulo (a NoSQL database) for geospatial indexing.
  • The "Wheel" Architecture: Analytics are designed to "plug in" to the Wheel. As new data arrives, it's processed by a series of analytics in a pipeline. Intermediate results are stored, and secondary analyses can be performed on these results.
  • Analytics: The paper details several specific analytics developed for the Wheel:
  • Contours and Clusters: Identifies spectrally anomalous regions in hyperspectral data.
  • Rare Pixel Finder (RPF): Detects small clusters of unusual pixels.
  • Gaussian Mixture Model and K-Nearest Neighbors (GMM-KNN): Another method for finding small clusters of unusual pixels.
  • Spectral Blobs: Uses a windowing technique to identify spectrally similar regions.
  • Supervised Spectral Classifier: A land cover classification algorithm, particularly useful for water and flood detection.

Datasets

The primary dataset used is daily hyperspectral satellite imagery from NASA's Earth Observing-1 (EO-1) satellite, specifically data from the Advanced Land Imager (ALI) and Hyperion instruments. The paper mentions using Level 1G data in GeoTIFF format. For the Supervised Spectral Classifier, training data was constructed from sections of EO-1 Hyperion images with known land coverage and cloud coverage.

Key Insights and Ideas

  • The "Scan Once, Analyze Many" Paradigm: The most significant insight is the efficiency gained by processing the entire dataset once for multiple scanning queries, rather than re-reading the data for each query. This is crucial for large-scale satellite imagery analysis.
  • Modularity and Extensibility: The "Wheel" architecture is designed to be highly modular, allowing new analytics to be easily integrated without modifying the core framework. This promotes a collaborative development environment.
  • Bridging Data and Decision-Makers: The framework not only processes raw data but also generates accessible visual reports and OGC-compliant Web Map Services, making the insights available to the public and decision-makers for applications like disaster relief.
  • Cloud-Native for Big Data: The reliance on OpenStack and Hadoop demonstrates a practical approach to leveraging cloud computing for handling the massive scale of Earth observation data.

Key Results

The paper describes the successful implementation of the Matsu Wheel for processing daily EO-1 data. The developed analytics have been used to identify interesting phenomena, such as volcanic events (e.g., Barren Island volcano). The Supervised Spectral Classifier showed promising results in distinguishing land cover types, with visual comparisons demonstrating consistency with existing methods, though some discrepancies in cloud coverage calculations were noted. The framework is designed to handle large datasets, with Hyperion scenes alone generating 1.5-2.5 GB of data per scene.

Limitations and Future Work

  • Limitations: The paper acknowledges that the Supervised Spectral Classifier's cloud coverage calculations may not perfectly align with USGS estimates, potentially due to differing treatments of thin cloud "haze." The performance of distributed systems can also be subject to variability due to shared workloads across virtual machines.
  • Future Work: The authors plan to extend the Supervised Spectral Classifier for specific applications like floodplain analysis for flood detection and developing fire detection analytics. They also aim to move beyond detection to prediction, proposing a framework for flood prediction that involves analyzing topological data, identifying flood basins, and correlating water coverage with water volume and rainfall. The goal is to continue developing the Matsu Wheel analytics and apply the framework to other Earth satellite datasets.

Summary updated

Abstract

Project Matsu is a collaboration between the Open Commons Consortium and NASA focused on developing open source technology for the cloud-based processing of Earth satellite imagery. A particular focus is the development of applications for detecting fires and floods to help support natural disaster detection and relief. Project Matsu has developed an open source cloud-based infrastructure to process, analyze, and reanalyze large collections of hyperspectral satellite image data using OpenStack, Hadoop, MapReduce, Storm and related technologies. We describe a framework for efficient analysis of large amounts of data called the Matsu "Wheel." The Matsu Wheel is currently used to process incoming hyperspectral satellite data produced daily by NASA's Earth Observing-1 (EO-1) satellite. The framework is designed to be able to support scanning queries using cloud computing applications, such as Hadoop and Accumulo. A scanning query processes all, or most of the data, in a database or data repository. We also describe our preliminary Wheel analytics, including an anomaly detector for rare spectral signatures or thermal anomalies in hyperspectral data and a land cover classifier that can be used for water and flood detection. Each of these analytics can generate visual reports accessible via the web for the public and interested decision makers. The resultant products of the analytics are also made accessible through an Open Geospatial Compliant (OGC)-compliant Web Map Service (WMS) for further distribution. The Matsu Wheel allows many shared data services to be performed together to efficiently use resources for processing hyperspectral satellite image data and other, e.g., large environmental datasets that may be analyzed for many purposes.
BibTeX
@inproceedings{patterson2016matsu,
  author = {Maria T. Patterson and Nikolas Anderson and Collin Bennett and Jacob Bruggemann and Robert L. Grossman and Matthew Handy and Vuong Ly and Daniel J. Mandl and Shane Pederson and James Pivarski and Ray Powell and Jonathan Spring and Walt Wells and John Xia},
  title = {{The Matsu Wheel: A Cloud-Based Framework for Efficient Analysis and Reanalysis of Earth Satellite Imagery}},
  booktitle = {BigDataService},
  year = {2016},
  url = {https://openreview.net/forum?id=1lOSJus5Zk},
}
Figure