Abstract
Copernicus is the European programme for monitoring the Earth. It consists of a set of systems that collect data from satellites and in-situ sensors, process this data and provide users with reliable and up-to-date information on a range of environmental and security issues. The data and information processed and disseminated puts Copernicus at the forefront of the big data paradigm, giving rise to all relevant challenges, the so-called 5 Vs: volume, velocity, variety, veracity and value. In this short paper, we discuss the challenges of extracting information and knowledge from huge archives of Copernicus data. We propose to achieve this by scale-out distributed deep learning techniques that run on very big clusters offering virtual machines and GPUs. We also discuss the challenges of achieving scalability in the management of the extreme volumes of information and knowledge extracted from Copernicus data. The envisioned scientific and technical work will be carried out in the context of the H2020 project ExtremeEarth which starts in January 2019.
Introduction
The Copernicus program represents a monumental achievement in Earth Observation (EO), generating petabytes of data through its constellation of Sentinel satellites. However, the sheer scale of this data has led to a "data rich but information poor" scenario. The ExtremeEarth project was launched to address this gap, aiming to revolutionize how we process, analyze, and extract actionable knowledge from these massive environmental datasets. By integrating distributed deep learning with semantic web technologies, ExtremeEarth seeks to democratize access to EO insights for critical global challenges.
The Big Data Challenge in Earth Observation
The Copernicus program faces the classic "5 Vs" of Big Data: volume, velocity, variety, veracity, and value. Beyond these standard metrics, the project identifies specific bottlenecks that hinder the full utilization of EO data:
- Accessibility: Current EO platforms often require specialized expertise, excluding a vast pool of software developers who could build downstream applications.
- Scalability Gaps: Existing geospatial database tools, such as RDF stores, typically fail when data exceeds the 100GB range. Copernicus, however, operates at the petabyte scale.
- Training Data Scarcity: Unlike general computer vision, which benefits from massive benchmarks like ImageNet, satellite imagery lacks large-scale, high-quality annotated datasets for training deep learning models.
Proposed Methodology: Extreme Earth Analytics
ExtremeEarth proposes a multi-pronged technical architecture to bridge these gaps, focusing on scale-out capabilities and semantic integration.
Distributed Deep Learning
To handle the spatial, spectral, and temporal complexity of Sentinel data, the project utilizes the HOPS data platform. This enables "scale-out" deep learning across large clusters of virtual machines and GPUs, moving beyond the limitations of single-node processing. This approach is specifically designed to handle the unique properties of Earth Observation data.
Extreme Linked Data and Semantic Catalogues
The project re-engineers semantic web tools like GeoTriples (for data transformation) and Strabon (for querying) to scale to trillions of metadata records. This allows for "Extreme Linked Data," where satellite imagery can be seamlessly queried alongside external datasets like OpenStreetMap. Furthermore, semantic catalogues will enable natural language-style queries, such as "How many icebergs were embedded in the Norske Øer Ice Barrier in 2017?", making the data accessible to non-experts.
Cloud Integration
By deploying these technologies directly on European cloud infrastructures like DIAS (Data and Information Access Services) and TEPs (Thematic Exploitation Platforms), ExtremeEarth moves the "processing to the data." This eliminates the need for users to download massive raw files, facilitating faster and more efficient analysis.
Data and Benchmarking
A core goal of the project is to contribute to the EO community's benchmarking resources. This includes leveraging the Eurosat benchmark (Sentinel-2 data with 13 spectral bands) and creating new "Extreme" datasets. The project aims to produce training datasets with millions of samples by expanding existing German Aerospace Center (DLR) datasets and incorporating crowdsourced data from OpenStreetMap.
High-Impact Use Cases
The project focuses on two primary "Challenges" to demonstrate its efficacy:
- Food Security: Producing 10-meter resolution water availability maps and automated crop boundary detection to support precision agriculture and sustainable farming.
- Polar Monitoring: Generating high-resolution (1km or better) sea ice concentration and type maps. This is critical for the safety of increased maritime traffic in the Arctic resulting from receding ice cover.
Conclusion
ExtremeEarth represents a significant shift toward "Extreme Earth Analytics." By addressing the scalability of both deep learning and metadata management, the project aims to unlock the full potential of the Copernicus program. While challenges remain in ground truth complexity and infrastructure maturity, the roadmap provides a clear path toward making Earth Observation data a standard utility for developers and decision-makers worldwide.
Summary updated