From Copernicus Big Data to Extreme Earth Analytics

Melanie Herschel, Héléna Galhardas, Berthold Reinwald, Irini Fundulaki, Carsten Binnig, Zoi Kaoudi, Manolis Koubarakis, Konstantina Bereta, Dimitris Bilidas, K. Giannousis, Theofilos Ioannidis, Despina Pantazi, George Stamoulis, Seif Haridi, Vladimir Vlassov, Lorenzo Bruzzone, Claudia Paris, Torbjørn Eltoft, Tanya Kramer, A. Charalabidis, Vangelis Karkaletsis, Stasinos Konstantopoulos, Jim Dowling, Theofilos Kakantousis, Mihai DatcuElib 2019 · DOI

Abstract

Copernicus is the European programme for monitoring the Earth. It consists of a set of systems that collect data from satellites and in-situ sensors, process this data and provide users with reliable and up-to-date information on a range of environmental and security issues. The data and information processed and disseminated puts Copernicus at the forefront of the big data paradigm, giving rise to all relevant challenges, the so-called 5 Vs: volume, velocity, variety, veracity and value. In this short paper, we discuss the challenges of extracting information and knowledge from huge archives of Copernicus data. We propose to achieve this by scale-out distributed deep learning techniques that run on very big clusters offering virtual machines and GPUs. We also discuss the challenges of achieving scalability in the management of the extreme volumes of information and knowledge extracted from Copernicus data. The envisioned scientific and technical work will be carried out in the context of the H2020 project ExtremeEarth which starts in January 2019.

Introduction

The Copernicus program represents a monumental achievement in Earth Observation (EO), generating petabytes of data through its constellation of Sentinel satellites. However, the sheer scale of this data has led to a "data rich but information poor" scenario. The ExtremeEarth project was launched to address this gap, aiming to revolutionize how we process, analyze, and extract actionable knowledge from these massive environmental datasets. By integrating distributed deep learning with semantic web technologies, ExtremeEarth seeks to democratize access to EO insights for critical global challenges.

The Big Data Challenge in Earth Observation

The Copernicus program faces the classic "5 Vs" of Big Data: volume, velocity, variety, veracity, and value. Beyond these standard metrics, the project identifies specific bottlenecks that hinder the full utilization of EO data:

  • Accessibility: Current EO platforms often require specialized expertise, excluding a vast pool of software developers who could build downstream applications.
  • Scalability Gaps: Existing geospatial database tools, such as RDF stores, typically fail when data exceeds the 100GB range. Copernicus, however, operates at the petabyte scale.
  • Training Data Scarcity: Unlike general computer vision, which benefits from massive benchmarks like ImageNet, satellite imagery lacks large-scale, high-quality annotated datasets for training deep learning models.

Proposed Methodology: Extreme Earth Analytics

ExtremeEarth proposes a multi-pronged technical architecture to bridge these gaps, focusing on scale-out capabilities and semantic integration.

Distributed Deep Learning

To handle the spatial, spectral, and temporal complexity of Sentinel data, the project utilizes the HOPS data platform. This enables "scale-out" deep learning across large clusters of virtual machines and GPUs, moving beyond the limitations of single-node processing. This approach is specifically designed to handle the unique properties of Earth Observation data.

Extreme Linked Data and Semantic Catalogues

The project re-engineers semantic web tools like GeoTriples (for data transformation) and Strabon (for querying) to scale to trillions of metadata records. This allows for "Extreme Linked Data," where satellite imagery can be seamlessly queried alongside external datasets like OpenStreetMap. Furthermore, semantic catalogues will enable natural language-style queries, such as "How many icebergs were embedded in the Norske Øer Ice Barrier in 2017?", making the data accessible to non-experts.

Cloud Integration

By deploying these technologies directly on European cloud infrastructures like DIAS (Data and Information Access Services) and TEPs (Thematic Exploitation Platforms), ExtremeEarth moves the "processing to the data." This eliminates the need for users to download massive raw files, facilitating faster and more efficient analysis.

Data and Benchmarking

A core goal of the project is to contribute to the EO community's benchmarking resources. This includes leveraging the Eurosat benchmark (Sentinel-2 data with 13 spectral bands) and creating new "Extreme" datasets. The project aims to produce training datasets with millions of samples by expanding existing German Aerospace Center (DLR) datasets and incorporating crowdsourced data from OpenStreetMap.

High-Impact Use Cases

The project focuses on two primary "Challenges" to demonstrate its efficacy:

  1. Food Security: Producing 10-meter resolution water availability maps and automated crop boundary detection to support precision agriculture and sustainable farming.
  2. Polar Monitoring: Generating high-resolution (1km or better) sea ice concentration and type maps. This is critical for the safety of increased maritime traffic in the Arctic resulting from receding ice cover.

Conclusion

ExtremeEarth represents a significant shift toward "Extreme Earth Analytics." By addressing the scalability of both deep learning and metadata management, the project aims to unlock the full potential of the Copernicus program. While challenges remain in ground truth complexity and infrastructure maturity, the roadmap provides a clear path toward making Earth Observation data a standard utility for developers and decision-makers worldwide.

Summary updated

Abstract

Copernicus is the European programme for monitoring the Earth. It consists of a set of systems that collect data from satellites and in-situ sensors, process this data and provide users with reliable and up-to-date information on a range of environmental and security issues. The data and information processed and disseminated puts Copernicus at the forefront of the big data paradigm, giving rise to all relevant challenges, the so-called 5 Vs: volume, velocity, variety, veracity and value. In this short paper, we discuss the challenges of extracting information and knowledge from huge archives of Copernicus data. We propose to achieve this by scale-out distributed deep learning techniques that run on very big clusters offering virtual machines and GPUs. We also discuss the challenges of achieving scalability in the management of the extreme volumes of information and knowledge extracted from Copernicus data. The envisioned scientific and technical work will be carried out in the context of the H2020 project ExtremeEarth which starts in January 2019
BibTeX
@inproceedings{herschel2019from,
  author = {Melanie Herschel and Héléna Galhardas and Berthold Reinwald and Irini Fundulaki and Carsten Binnig and Zoi Kaoudi and Manolis Koubarakis and Konstantina Bereta and Dimitris Bilidas and K. Giannousis and Theofilos Ioannidis and Despina Pantazi and George Stamoulis and Seif Haridi and Vladimir Vlassov and Lorenzo Bruzzone and Claudia Paris and Torbjørn Eltoft and Tanya Kramer and A. Charalabidis and Vangelis Karkaletsis and Stasinos Konstantopoulos and Jim Dowling and Theofilos Kakantousis and Mihai Datcu},
  title = {{From Copernicus Big Data to Extreme Earth Analytics}},
  booktitle = {Elib},
  year = {2019},
  doi = {10.5441/002/edbt.2019.88},
}
Figure