Semantics First, Pixels Later: Mission-Driven Event Streams for Drone Video over Bandwidth-Starved Links

Cong XuarXiv 2026 · arXiv

TL;DR

This paper introduces BPS (Behavior-Primitive Stream), a communication paradigm for drones that prioritizes symbolic event records over raw pixels. By sending semantic "behavior primitives" (like "people converging") instead of video frames, the system achieves mission-critical question-answering accuracy at bitrates as low as 0.13 kbps—matching the performance of standard H.265 video at roughly 800 times the bandwidth.

What Problem Does This Paper Address?

In tactical, search-and-rescue, or remote surveillance scenarios, drones often operate over "bandwidth-starved" links—tactical radios or satellite uplinks that provide only a few kilobits per second. Standard video codecs like H.265 are designed for human eyes; at extremely low bitrates, they produce "pixel soup" that is useless for decision-making.

The authors argue that operators rarely need the pixels themselves. Instead, they need answers to specific questions: Did the vehicles stop? Did a crowd form? When did the area become active? The paper tackles the challenge of how to deliver these answers reliably when the link is too thin to support even a low-resolution video stream.

Methods

The authors propose a "Semantics First, Pixels Later" architecture called BPS. The core components include:

  1. Onboard Perception: The drone runs a real-time detector (RF-DETR-M) and tracker (OC-SORT) locally.
  2. Behavior Primitives: A streaming engine transforms raw tracks into high-level events. The paper defines seven primitives: dwell, density change, birth after quiet, flow deviation, approach (convergence), persistent change, and access-rate change.
  3. Mission Cards: These are small JSON-based "standing orders" (16–125 bytes) sent to the drone that define which primitives, object classes, and regions the operator cares about.
  4. Prioritized Scheduler: A custom scheduler ensures that event records (averaging just 25 bytes) are sent first. If bandwidth remains, it sends quantized positions, then small image patches, and finally full keyframes.
  5. LLM-Based Evaluation: To measure success, the authors use a Large Language Model (Olmo-3-7B) as the "reader." They compare how well the LLM can answer multiple-choice questions about the scene when given the BPS text stream versus when its vision-tuned counterpart (Molmo) views H.265 video.

Datasets

The study utilizes two primary aerial video datasets:

  • VisDrone2019-VID: 80 sequences featuring hover and moving cameras.
  • Okutama-Action: 43 sequences of a moving camera over a sports ground, used primarily for development.

Because no existing benchmark asks "where, when, and how many" for multi-frame events, the authors generated a new set of 1,638 questions derived from annotated ground-truth tracks to serve as a rigorous evaluation framework.

Key Insights and Ideas

  • The Semantic Advantage: Semantics are incredibly dense. An event record is orders of magnitude smaller than a single compressed video frame but contains the specific information needed to answer an operator's query.
  • Graceful Degradation: Unlike video, which becomes unwatchable when packets are lost or bandwidth drops, BPS is designed to "shed" pixels and positions while keeping the semantic event stream alive down to 0.3 kbps.
  • The Perception Bottleneck: The study reveals that while a "perfect perception" oracle would dominate video, real-world onboard perception is the weak link. Errors in detection and tracking significantly degrade the utility of the semantic stream.
  • LLM as a Unified Interface: Using an LLM to "read" the stream allows for a flexible, natural language interface for the operator, effectively turning a stream of bytes into a searchable, queryable log of behaviors.

Key Results

  • Efficiency: On the development set, the BPS event stream at 0.13 kbps answered 74.3% of questions. To get comparable answers from video using ground-side perception, a link of 100 kbps was required—an 800x difference in efficiency.
  • Timeliness: Event records arrive on the ground within 0.5 seconds of the drone deciding an event has occurred, even on 10 kbps links. In contrast, low-latency H.265 frames at that same bitrate were often over 5 seconds old.
  • Resilience: The event stream maintained high accuracy even with 20% packet loss, as the records are small enough to be sent frequently and independently.
  • Hardware Constraints: Running the full pipeline on a Jetson Orin Nano resulted in a frame rate of 4.5 fps. While this caused the system to miss some fast-moving events, the accuracy remained higher than the H.265 baseline.

Limitations and Future Work

  • Narrow Event Scope: 84% of the development questions focused on "convergence" (approach). The performance on more complex or diverse behaviors needs further validation.
  • Perception Sensitivity: The system is highly dependent on the quality of the onboard detector. The authors note that the pipeline "misses most events" on low-power embedded hardware, suggesting a need for lighter, more specialized models.
  • Human-in-the-loop: The "mission cards" were drafted by LLMs for the study; future work should explore how human operators interact with and refine these cards in real-time.
  • Oracle vs. Reality: There is a significant gap between "perfect" annotated tracks and "real" onboard perception. Closing this gap is the primary hurdle for making BPS a production-ready reality.

Summary updated

Abstract

A surveillance drone on a bandwidth-starved link cannot send the video its operator would need, yet the operator's questions are often not about pixels: did three people converge, when, and where? We study what should cross a link of 1 Mbps down to 1 kbps instead, and in what order. BPS, a behavior-primitive stream, sends semantics first and pixels later: it runs a detector and a tracker onboard and sends short records (25 bytes on average) of multi-frame events selected by a mission card of 16-125 bytes, ahead of positions and pictures. We measure it by what a ground-side language model answers from it, under a preregistered protocol with a held-out set, against H.265 read by the vision fine-tune of the same model, H.265 on which the ground runs our own perception, a stream of quantized tracks, and onboard captions. On the development set (84% of questions about convergence, mostly one location) the event stream answers 74.3% of questions at 0.13 kbps. A reader that receives nothing answers 57.4%, and the video reader 59.6% at 100 kbps and 64.7% on the source frames. When the ground instead runs our perception on the video, video matches the stream at 100 kbps, about 800 times the stream's rate, and falls behind at 64 kbps and below. A track stream matches it at 96 times the bytes and loses 0.03 at 17 times. On held-out data the result is weaker: with events taken from annotated tracks, an oracle, the stream beats H.265 at 100 kbps (+0.13), but with onboard perception it does not (+0.01 [-0.04, +0.06]) and is no better than receiving nothing. Sent alone through emulated links the event records answer 0.739 at 1 Mbps and still 0.724 at 0.3 kbps; an every-frame track stream holds down to 1 kbps and video with ground-side perception down to about 100 kbps. On a Jetson Orin Nano the unchanged pipeline runs at 4.5 fps and misses most events.
BibTeX
@misc{xu2026semantics,
  author = {Cong Xu},
  title = {{Semantics First, Pixels Later: Mission-Driven Event Streams for Drone Video over Bandwidth-Starved Links}},
  year = {2026},
  eprint = {2611.19463},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19463},
}
Figure