Sensor Artificial Intelligence and its Application to Space Systems - A White Paper

Anko Börner, Heinz-Wilhelm Hübers, Odej Kao, Florian Schmidt, Sören Becker, Joachim Denzler, Daniel Matolin, David Haber, Sergio Lucia, Wojciech Samek, Rudolph Triebel, Sascha Eichstädt, Felix Bießmann, Anna M. Kruspe, Peter Jung, Manon Kok, Guillermo Gallego, Ralf BergerarXiv 2020 · OpenReview

Sensor AI White Paper Banner

Introduction

This white paper introduces "Sensor AI," a paradigm shift that integrates physical sensor models and "digital twins" directly into machine learning pipelines. By bridging the gap between abstract data science and physical reality, the authors argue for the creation of robust, efficient, and interpretable AI necessary for high-stakes environments like autonomous space exploration.

The Challenge: Trust and Hardware Constraints

Modern AI has seen incredible success in general applications, but it often treats data as an abstract starting point, ignoring the physical processes that generated it. This "black box" approach creates two major hurdles for safety-critical fields:

  1. Lack of Trust: Without understanding how physical variables (like temperature or radiation) affect sensor data, it is difficult to provide the quality guarantees required for medical, transport, or space applications.
  2. The Space Constraints Gap: Space missions face a brutal hardware reality. While modern sensors generate gigabits of data per second, onboard CPUs are often decades old and power-constrained to a few dozen watts. We cannot simply "throw a GPU at it." We need a way to process and compress data by a factor of 100x or more onboard, which requires AI that "understands" the sensor it's attached to.

The Sensor AI Paradigm

The paper proposes Sensor AI as an interdisciplinary research field that merges sensor physics, communication networks, and artificial intelligence. Rather than standard "data-in, prediction-out" models, Sensor AI advocates for:

  • Physical Grounding: Linking digital data to SI units by incorporating detailed physical models (digital twins) of the sensors into the learning process.
  • Hybrid Modeling: Using physical models to support data-driven models (and vice versa) to estimate uncertainties and improve robustness.
  • Onboard Semantic Processing: Moving away from simple data compression toward high-level semantic processing on the satellite itself, allowing only the most relevant information to be downlinked.
  • Hardware-Aware Design: Developing AI methods specifically optimized for the limited, radiation-hardened embedded systems found in space.

Key Insights

  • The "Missing Element": The authors argue that the sensor itself is the missing link in the current AI process chain. By ignoring the circumstances of data collection (e.g., how a sensor's noise profile changes under radiation), we limit the reliability of the AI.
  • Space as the Ultimate Stress Test: Astronautics is presented as the perfect framework for Sensor AI because it combines extreme environmental loads (vacuum, zero-G, radiation) with severely limited computational resources.
  • Closing the Loop: Sensor AI isn't just about better data analysis; it’s about using AI to improve the sensors themselves—from design and characterization in the lab to predictive control during a mission.

Requirements for Space Deployment

As a white paper, the document focuses on defining requirements and research trajectories rather than quantitative experimental results. However, it establishes several critical benchmarks for success in the field:

  • Data Reduction: The necessity of achieving a 100x or greater reduction in data volume through onboard processing.
  • Power Efficiency: The requirement for AI to run on processing units consuming only ~10W, as high-power GPUs are non-starters for space missions.
  • Standardization: The paper highlights the need for AI to meet the strict development standards of space hardware and software, where "updates in orbit" are often limited or impossible.

Future Directions

The authors explicitly list research questions that define the future of the field, focusing on how AI can learn insights about the sensor itself and the trade-offs between data compression and model interpretability. The primary challenge remains the current "black box" nature of deep learning, which must be overcome to meet the transparency and technical robustness required for trustworthy AI in safety-critical deployments.

Summary updated

Abstract

Information and communication technologies have accompanied our everyday life for years. A steadily increasing number of computers, cameras, mobile devices, etc. generate more and more data, but at the same time we realize that the data can only partially be analyzed with classical approaches. The research and development of methods based on artificial intelligence (AI) made enormous progress in the area of interpretability of data in recent years. With growing experience, both, the potential and limitations of these new technologies are increasingly better understood. Typically, AI approaches start with the data from which information and directions for action are derived. However, the circumstances under which such data are collected and how they change over time are rarely considered. A closer look at the sensors and their physical properties within AI approaches will lead to more robust and widely applicable algorithms. This holistic approach which considers entire signal chains from the origin to a data product, "Sensor AI", is a highly relevant topic with great potential. It will play a decisive role in autonomous driving as well as in areas of automated production, predictive maintenance or space research. The goal of this white paper is to establish "Sensor AI" as a dedicated research topic. We want to exchange knowledge on the current state-of-the-art on Sensor AI, to identify synergies among research groups and thus boost the collaboration in this key technology for science and industry.
BibTeX
@misc{borner2020sensor,
  author = {Anko Börner and Heinz-Wilhelm Hübers and Odej Kao and Florian Schmidt and Sören Becker and Joachim Denzler and Daniel Matolin and David Haber and Sergio Lucia and Wojciech Samek and Rudolph Triebel and Sascha Eichstädt and Felix Bießmann and Anna M. Kruspe and Peter Jung and Manon Kok and Guillermo Gallego and Ralf Berger},
  title = {{Sensor Artificial Intelligence and its Application to Space Systems - A White Paper}},
  year = {2020},
  eprint = {2006.08368},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2006.08368},
}
Figure