Abstract Project Matsu is a collaboration between the Open Commons Consortium and NASA focused on developing open source technology for the cloud-based processing of Earth satellite imagery. A particular focus is the development of applications for detecting fires and floods to help support natural disaster detection and relief. Project Matsu has developed an open source cloud-based infrastructure to process, analyze, and reanalyze large collections of hyperspectral satellite image data using OpenStack, Hadoop, MapReduce, Storm and related technologies. We describe a framework for efficient analysis of large amounts of data called the Matsu "Wheel." The Matsu Wheel is currently used to process incoming hyperspectral satellite data produced daily by NASA's Earth Observing-1 (EO-1) satellite. The framework is designed to be able to support scanning queries using cloud computing applications, such as Hadoop and Accumulo. A scanning query processes all, or most of the data, in a database or data repository. We also describe our preliminary Wheel analytics, including an anomaly detector for rare spectral signatures or thermal anomalies in hyperspectral data and a land cover classifier that can be used for water and flood detection. Each of these analytics can generate visual reports accessible via the web for the public and interested decision makers. The resultant products of the analytics are also made accessible through an Open Geospatial Compliant (OGC)-compliant Web Map Service (WMS) for further distribution. The Matsu Wheel allows many shared data services to be performed together to efficiently use resources for processing hyperspectral satellite image data and other, e.g., large environmental datasets that may be analyzed for many purposes.
Introduction
As the cost of storage and processing drops, scientists are increasingly managing high-resolution satellite imagery at a scale that challenges traditional analysis methods. The primary bottleneck to discovery is no longer a lack of data, but the inability to efficiently manage and analyze it. Specifically, researchers often need to run "scanning queries"—computations applied across an entire massive dataset, such as searching every image in a repository for a specific spectral signature. If a distributed system has to retrieve and scan the entire dataset from scratch for every single query, the infrastructure quickly becomes overloaded. The Matsu Wheel was developed to address this by running multiple, simultaneous scanning queries over high-volume hyperspectral satellite data without crippling network bandwidth or storage systems.
The Matsu Wheel Framework
The Matsu Wheel is a framework built on the Open Science Data Cloud (OSDC) using OpenStack and Hadoop (HDFS, MapReduce, Accumulo). The core methodological shift is the "Wheel" architecture. Instead of querying data in a traditional sense, the data flows through the system once—like pavement passing under a bicycle wheel—while multiple analytics sit in place and process the data chunks as they arrive. This means the heavy lifting of data retrieval and pre-processing is only done once, regardless of how many different algorithms are applied to it.
Several plug-and-play analytics are currently running on the Wheel:
- Contours and Clusters: Uses Principal Component Analysis (PCA) and k-means clustering to find geographically grouped pixels with highly anomalous spectral signatures.
- Rare Pixel Finder & GMM-KNN: Algorithms utilizing Mahalanobis distance, Gaussian Mixture Models, and K-Nearest Neighbors to isolate small clusters of unusual pixels.
- Supervised Spectral Classifier: A Support Vector Machine (SVM) trained to classify every pixel in an image into categories including Clouds, Water, Desert/Dry Land, or Vegetation.
Datasets and Validation
The primary data source is NASA’s Earth Observing-1 (EO-1) satellite, specifically utilizing the Advanced Land Imager (ALI) and the Hyperion hyperspectral imager, which captures 242 spectral bands. The system processes Level 1G GeoTiff images. For the SVM classifier, a training set of 6,000 to 9,000 objects per class was extracted from visually confirmed Hyperion scenes. Results were validated against USGS EarthExplorer benchmarks to ensure accuracy in cloud and land cover calculations.
Key Results
The Matsu Wheel successfully processes the daily influx of hyperspectral data from NASA's EO-1 satellite, converting raw radiances to reflectances and running five distinct analytics in parallel. A notable operational success occurred in February 2014, when the Contours and Clusters analytic automatically flagged highly anomalous spectral activity. Upon review, this was confirmed to be an active eruption of the Barren Island volcano.
Furthermore, the SVM-based Supervised Spectral Classifier successfully segments complex coastal and island scenes. When compared to USGS EarthExplorer benchmarks for 20 random test scenes, the Wheel's cloud and water coverage calculations showed a strong, consistent linear relationship with the expected USGS estimates.
Conclusion and Future Work
By decoupling data retrieval from analytics, the Matsu Wheel avoids the severe I/O bottlenecks that usually plague distributed databases during full-dataset scans. While the current SVM classifier faces minor challenges with shadowed areas and thin cloud haze, the framework remains highly extensible. Future work will shift from detection toward prediction, specifically for flood disaster relief, by integrating topological data, historical water coverage, and global precipitation data.
Summary updated