Predicting the Needle in a Petabyte Scale Haystack: Open-Vocabulary Event Anticipation in Satellite Imagery

Shreelekha Revankar, Mikhail Klassen, Creon Levit, Ash Hoover, Kavita Bala, Bharath HariharanarXiv 2026 · arXiv

TL;DR

Researchers have developed an automated "data engine" that discovers and labels global events in satellite imagery without manual intervention by treating change as vector arithmetic in a vision-language latent space. Using this data, they trained a causal transformer model that can detect ongoing changes, predict their future states, and forecast completion dates with a 3-day median error—up to nine months before the event finishes.

What Problem Does This Paper Address?

Satellite constellations like PlanetScope and Sentinel-2 generate petabytes of data daily, capturing the entire lifecycle of human and natural events, from urban sprawl to tropical deforestation. However, significant events are "needles in a haystack"—most of the Earth remains stable most of the time.

Existing satellite analysis systems face three major hurdles:

  1. The Data Bottleneck: Manually annotating events at a global scale is impossible, and existing datasets are geographically limited or restricted to a few categories.
  2. Joint Identification: Current models struggle to simultaneously answer where a change is happening, what it will become, and when it will be done.
  3. Open-Vocabulary Constraints: Most systems are trained on a fixed set of labels (e.g., "building" vs. "no building") and cannot handle arbitrary natural language queries about new types of events.

Methods

The authors propose a two-part solution: an automated data discovery pipeline and a latent-space "world model" for anticipation.

1. The Automated Data Engine

The core insight is that while we lack open-vocabulary event detectors, we have excellent static Vision-Language Models (VLMs). The researchers leverage latent space arithmetic: they represent a change event as the vector difference between embeddings at two time points ($\Delta e = e_{after} - e_{before}$).

To build their dataset, they use a Large Language Model (LLM) to generate "before" and "after" text descriptions for diverse events (e.g., "open empty field" $\rightarrow$ "grid of dark rectangular solar panels"). They then use two strategies to find these events in the archives:

  • Intersection Search: Finding locations that match the "before" description at time $t_1$ and the "after" description at $t_2$.
  • Change Vector Search: Searching directly for the semantic transformation ($\Delta e$) in the embedding space.

They combine these using Automated Relevance Feedback (ARF), where a multimodal LLM (like Gemini) verifies initial candidates to create a refined "change probe" for better retrieval.

Figure 2: Change-Aware Retrieval and Automated Relevance Feedback. (Left) Intersection search enforces scene context at both timepoints but can produce false positives when a scene already matches the after-state without change. Change vector search finds the correct transformation but often loses context. (Right) ARF connects both: an MLLM scores initial intersection results, confirmed positives form averaged probe ∆eprobe for refined re-retrieval.

2. The Anticipation Model

Once they have a massive dataset of events, they train a Satellite Event Anticipator. This is a causal transformer that operates entirely within the embedding space of a frozen VLM (RS-SigLIP). It ingests a sequence of historical observations (trajectories in latent space) and a target time horizon ($h$) to predict:

  1. The future embedding ($e_{pred}$) at $h$ days.
  2. The time-to-completion ($t_{pred-comp}$) for the event.

By operating in the latent space rather than pixel space, the model is significantly more robust to common satellite imagery noise like clouds, haze, and sensor artifacts.

Datasets

The authors introduced SatEvents-21k, the largest global satellite event dataset to date.

  • Scale: 21,000+ unique locations across six continents.
  • Diversity: 64+ positive event classes (construction, wildfires, etc.) and 69 stable "negative" scene categories.
  • Imagery: ~2 million images from PlanetScope (daily, 3m resolution) and Sentinel-2 (5-day, 10m resolution).
  • Availability: The authors intend to release this dataset publicly.

Key Insights and Ideas

  • Change as Arithmetic: The most powerful idea is that semantic change can be captured by simple vector subtraction in a well-aligned vision-language space. This bypasses the need for manual labels.
  • Latent-Space World Modeling: Predicting future embeddings rather than future pixels avoids the computational cost and noise sensitivity of image generation while retaining the semantic information needed for "open-vocabulary" queries.
  • Zero-Shot Anticipation: Because the model is grounded in a VLM, it can anticipate events it has never seen during training (e.g., predicting a "bridge construction" event even if it was only trained on "highway construction").
  • Predictability varies by event type: Large-scale infrastructure (airports, dams) is detectable up to a year before completion, whereas rapid natural events (wildfires, floods) offer much shorter anticipation windows.
Figure 3: Earliest detection horizons vary significantly by event type, with infrastructure being predictable much earlier than natural disasters.

Key Results

The model significantly outperforms existing baselines (including frontier MLLMs like Gemini 3 Flash and specialized satellite models like TeoChat):

  • Change Detection: Achieved a 93.5% max F1 score, effectively filtering out stable locations at a global scale.
  • Event Retrieval: Outperformed baselines by 3.6x on predicted event retrieval.
  • Anticipation Horizon: Maintained strong predictive signals even 9+ months before event completion, whereas other models' performance collapsed. (See Figure 6 below).
  • Completion Accuracy: Forecasted completion dates with a 3-day median error on PlanetScope imagery.
Figure 6: Retrieval performance (mAP@10) by lead time, showing the model's stability over long horizons compared to baselines.

Limitations and Future Work

  • Single Event Assumption: The current data engine assumes one dominant event between two time points, potentially missing complex, multi-stage transitions (e.g., empty lot $\rightarrow$ parking lot $\rightarrow$ mall).
  • VLM Capping: The model's "intelligence" is ultimately capped by the underlying VLM's vocabulary; if the VLM can't recognize a concept, the anticipation model won't either.
  • Ambiguity of "Completion": Defining exactly when an event "finishes" is subjective (e.g., is a building done when the roof is on, or when the parking lot is paved?). The authors noted higher errors in categories where the end-state is visually ambiguous.
  • Future Work: Scaling the engine to multi-year cadences and exploring fine-tuning the underlying VLM on the newly discovered satellite-specific image-text pairs.

Summary updated

Abstract

Satellite constellations now image the Earth daily, capturing the lifecycle of events from deforestation to urban construction. Anticipating these events before completion enables timely intervention, yet existing systems cannot jointly identify where a change occurs, what it will become, and when it will finish across an open vocabulary. Building such a model requires diverse event data, but events are like needles in a haystack, making manual annotation at global scale infeasible. We address this problem by representing change as the vector difference between vision-language model embeddings at distinct time points. This arithmetic approach allows us to search for semantic transformations directly in the latent space, enabling (1) an automated data engine for global event discovery without manual annotation and (2) the largest global satellite event dataset to our knowledge, comprising 21,000+ locations across 64+ event types with ~2M images from PlanetScope and Sentinel-2 spanning six continents, to train (3) a novel global-scale event anticipation model. Our model detects change with a 93.5% max F1 score, outperforms baselines 3.6x on predicted event retrieval, and forecasts completion dates with a 3-day median error, even nine months before completion. Code: https://github.com/ShreelekhaR/predicting-the-needle
BibTeX
@misc{revankar2026predicting,
  author = {Shreelekha Revankar and Mikhail Klassen and Creon Levit and Ash Hoover and Kavita Bala and Bharath Hariharan},
  title = {{Predicting the Needle in a Petabyte Scale Haystack: Open-Vocabulary Event Anticipation in Satellite Imagery}},
  year = {2026},
  eprint = {2611.16000},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16000},
}
Figure