Fake or Real: The Impostor Hunt in Texts for Space Operations

Agata Kaczmarek, Dawid Pludowski, Piotr Wilczynski, Krzysztof Kotowski, Ramez Shendy, Evridiki Vasileia Ntagiou, Jakub Nalepa, Artur Janicki, Przemyslaw BiecekarXiv 2025 · DOI · arXiv

Abstract

This paper presents the framework for the "Fake or Real" Kaggle competition, an initiative by the European Space Agency (ESA) in collaboration with the Warsaw University of Technology and KP Labs. The challenge addresses the growing security risks of integrating Large Language Models (LLMs) into mission-critical space environments. Unlike traditional AI detection tasks that focus on human-authored versus machine-generated text, this competition requires participants to distinguish between "proper" LLM outputs and those generated by models that have been maliciously modified or "poisoned." By focusing on an "AI vs. AI" detection paradigm, the challenge aims to develop robust verification methods for environments where model weights are inaccessible and reliability is paramount.

Introduction

As LLMs are increasingly integrated into industrial workflows, the space sector is exploring their utility for high-stakes tasks such as assisting mission controllers and investigating satellite anomalies. However, space operations demand extreme reliability and must adhere to strict European Cooperation for Space Standardization (ECSS) guidelines. The authors identify two primary threats to this reliability: data poisoning, where malicious actors corrupt training data to induce biased or "Trojan" responses, and user overreliance, where subtle hallucinations go unnoticed.

Because most organizations utilize LLMs as "black boxes" (accessing them via APIs without visibility into underlying weights), there is an urgent need for methods that can detect manipulated content based solely on the generated text.

Data Generation and Methodology

The competition utilizes a specialized dataset derived from The Messenger, a journal published by the European Southern Observatory (ESO) covering 50 years of astronomy and space technology. The dataset consists of pairs of summaries generated from real scientific articles.

The data generation pipeline for the 'Fake or Real' competition, illustrating the creation of proper and malicious summary pairs.

As illustrated in the generation pipeline, for each source article, two summaries are produced using the same LLM. One summary is generated by a properly functioning model, while the other is produced by a model that has been maliciously altered through undisclosed techniques such as weight manipulation or prompt injection. These summaries are then randomly shuffled, and participants must provide a binary label to identify the benign output.

Baseline Detection Methods

The paper provides two baseline methods for participants:

  • Word-Level Language Detection: Utilizing libraries like langdetect to identify the ratio of English to non-English words, assuming some attacks cause the model to "leak" foreign language fragments.
  • Character-Level Analysis: Evaluating the ratio of Latin characters to total characters to catch "noisy" hallucinations involving unexpected Unicode symbols or emojis.

Key Insights and Results

The competition shifts the research focus from "AI vs. Human" to "AI vs. AI," mimicking real-world scenarios where engineers must verify outputs from third-party APIs. The "fake" texts in the dataset are not always obvious gibberish; they range from subtle factual shifts (e.g., introducing irrelevant topics like dinosaurs into space articles) to stylistic changes and technical noise.

Initial results indicate high engagement, with over 800 entrants and 1,500 submissions within the first 20 days. While simple character-level rules can identify noisy fakes, they fail against coherent but factually incorrect text. This highlights the necessity for advanced Natural Language Processing (NLP) techniques—such as sentiment analysis, part-of-speech (POS) tagging, and semantic consistency checks—to achieve reliable detection.

Conclusion and Future Work

The specific attack methods used to generate malicious summaries remain undisclosed to prevent participants from over-fitting to specific techniques. Future work will involve a meta-analysis of the winning strategies, which will be presented at the ESA AI STAR conference in late 2025. These findings are expected to establish a new state-of-the-art framework for AI assurance and security within the space domain.

Summary updated

BibTeX
@misc{kaczmarek2025fake,
  author = {Agata Kaczmarek and Dawid Pludowski and Piotr Wilczynski and Krzysztof Kotowski and Ramez Shendy and Evridiki Vasileia Ntagiou and Jakub Nalepa and Artur Janicki and Przemyslaw Biecek},
  title = {{Fake or Real: The Impostor Hunt in Texts for Space Operations}},
  year = {2025},
  eprint = {2507.13508},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.13508},
}
Figure