AVERY: Intent-Driven Adaptive VLM Split Computing via Embodied Self-Awareness for Efficient Disaster Response Systems

Rajat Bhattacharjya, Sing-Yao Wu, Hyunwoo Oh, Chaewon Nam, Suyeon Koo, Mohsen Imani, Elaheh Bozorgzadeh, Nikil DuttarXiv 2025 · arXiv

TL;DR

AVERY is an adaptive split computing framework that enables resource-constrained drones to deploy powerful Vision-Language Models (VLMs) for disaster response. By organizing computation into a cognitive-inspired dual-stream architecture (fast context vs. deep insight) and using a self-aware onboard controller, it dynamically balances operator intent, network bandwidth, and energy consumption.

What Problem Does This Paper Address?

Drones (UAVs) are vital tools in disaster response scenarios like floods and wildfires. However, traditional onboard computer vision pipelines rely on rigid Convolutional Neural Networks (CNNs) that generate generic labels and completely lack the semantic flexibility to answer complex, natural-language queries from operators (e.g., "locate vehicles partially submerged but still accessible by road").

While modern Vision-Language Models (VLMs) possess this advanced reasoning capability, they are far too computationally heavy and power-hungry to run entirely on a drone's limited battery. Conversely, naively streaming raw video data to a cloud server fails because disaster zones typically suffer from highly unstable, low-bandwidth network connectivity. AVERY addresses this dilemma by asking: How can we enable intent-driven, semantically rich, and energy-efficient VLM inference on drones operating in communication-compromised environments?

Methods

AVERY introduces a two-level hierarchical adaptive split computing strategy. Instead of simply slicing a model at a arbitrary depth based on network conditions, AVERY first splits the workload by semantic function using a dual-stream architecture inspired by human visual cognition:

  1. Context Stream: Powered by a lightweight CLIP encoder, this stream runs continuously at a high frequency. It provides low-resolution, text-only reasoning to maintain fast, real-time situational awareness for the operator.
  2. Insight Stream: Triggered only when needed, this stream uses a heavier Segment Anything Model (SAM) vision backbone combined with CLIP features to deliver low-frequency, high-fidelity spatial grounding (such as precise pixel-level segmentation masks).
Fig. 1. The Motivation and AVERY Paradigm. (a) Conventional CNNs cannot support the distinct, prompt-based reasoning needed for multi-level disaster response, such as broad Context queries for triage and specific Insight queries for investigation. (b) While a full on-device VLM can process these queries, its prohibitive energy cost makes it infeasible for UAV deployment. (c) Naive cloud offloading fails under the unreliable, low-bandwidth networks found in disaster zones. (d) AVERY resolves these conflicts with a dual-stream (context and insight) split computing architecture, intelligently transmitting either lightweight context features or compressed insight activations to enable hierarchical, efficient, multi-level intelligence.

An onboard, self-aware controller manages this hierarchy at runtime:

  • Level 1 (Intent Gating): The controller parses the operator's natural language query. If it is a broad triage question ("Is anyone on the roof?"), it routes execution through the cheap Context Stream. If it requires precise localization ("Highlight trapped vehicles"), it activates the Insight Stream.
  • Level 2 (Resource Adaptation): When the heavy Insight Stream is active, the controller splits the model early—right after the first Vision Transformer (ViT) block—and applies learned bottleneck compression to the intermediate activations. It then consults a pre-profiled Lookup Table (LUT) to dynamically switch between High Accuracy, Balanced, and High Throughput operating tiers based on current network bandwidth and power budgets.
AVERY Architecture Overview

Datasets

To train and evaluate the system in a realistic disaster context, the authors curated and released Flood-ReasonSeg. This dataset consists of flood-scene images annotated with natural-language instructions and precise segmentation masks targeting two critical classes: stranded individuals and vehicles trapped by water. The authors used this dataset alongside Low-Rank Adaptation (LoRA) to fine-tune a flood-specialized variant of the LISA VLM framework.

Key Insights and Ideas

  • Intent as a First-Class Constraint: Traditional split computing treats partitioning purely as a resource-optimization problem (balancing latency vs. data size). AVERY demonstrates that for VLMs, the split must first be driven by what semantic product the human operator actually needs.
  • The "Sweet Spot" Split Point: Detailed profiling revealed that splitting the SAM vision backbone immediately after the first ViT block (split@1) serves as an incredibly powerful feature extractor. Compressing data at this stage yields a massive drop in edge energy consumption while retaining enough distilled information for the server to reconstruct high-quality segmentation masks. Deeper splits actually degrade accuracy initially while sharply increasing edge latency.
  • Cognitive-Inspired Efficiency: Separating rapid, coarse awareness from slower, deep semantic understanding mimics human vision pathways. This prevents the drone from wasting precious energy over-provisioning resources for simple queries.

Key Results

The framework was evaluated using a LISA-7B model deployed across an NVIDIA Jetson AGX Xavier (simulating the drone) and an NVIDIA RTX 6000 Ada GPU (the remote server) under fluctuating network conditions (8–20 Mbps).

  • Massive Energy Savings: AVERY reduced onboard energy consumption by 93.98% compared to executing the full SAM backbone locally on the drone.
  • High Accuracy Retention: It achieved 11.2% higher accuracy than raw image compression baselines. During dynamic network adaptation, its average accuracy remained within 0.75% of the ideal, static High-Accuracy baseline.
  • Stable Throughput: While a static high-accuracy setup completely collapses when bandwidth drops, AVERY's adaptive logic maintained a stable throughput of 0.74 Packets Per Second (PPS) in accuracy mode and up to 1.85 PPS when prioritizing throughput.

Limitations and Future Work

While AVERY marks a significant step forward, the authors note several areas for improvement. The current prototype operates under a fixed onboard compute power mode, meaning future iterations must account for fluctuating drone power envelopes. Additionally, the controller relies on a deterministic lookup table; the authors suggest that incorporating advanced control policies like Reinforcement Learning (RL) or Model Predictive Control (MPC) could better exploit complex environmental signals. Finally, extending this framework from a single drone to multi-UAV coordinated swarms remains an open challenge.

Summary updated

Abstract

Unmanned Aerial Vehicles (UAVs) in disaster response require complex, queryable intelligence that onboard CNNs cannot provide. While Vision-Language Models (VLMs) offer this semantic reasoning, their high resource demands make on-device deployment infeasible, and naive cloud offloading fails under the low-bandwidth, unstable networks endemic to disaster zones. We present AVERY, an intent-driven adaptive split computing framework for efficient VLM deployment on resource-constrained platforms. AVERY is motivated by the observation that operator intent must be treated as a first-class system objective, since missions such as broad situational monitoring and precise, spatially grounded investigation require different semantic products, latency targets, and resource allocations. To reflect this, AVERY advances split computing beyond traditional depth-wise partitioning through a functional, cognitive-inspired dual-stream split: a high-frequency, low-resolution Context stream for real-time awareness, and a low-frequency, high-fidelity Insight stream for deep analysis. This design enables a hierarchical split strategy: computation is first separated by function, then partitioned depth-wise across edge and cloud when the Insight stream is required. A lightweight, self-aware onboard controller monitors network conditions and operator intent to select from pre-trained compression models, navigating the accuracy-throughput trade-off at runtime. Evaluated using LISA-7B in an edge-cloud setting under fluctuating network conditions, AVERY achieves 11.2% higher accuracy than raw image compression, 93.98% lower energy consumption than full-edge execution, and average accuracy within 0.75% of the static High-Accuracy baseline during dynamic adaptation. Overall, AVERY enhances mission efficiency and enables real-time, queryable intelligence in dynamic disaster environments.
BibTeX
@misc{bhattacharjya2025avery,
  author = {Rajat Bhattacharjya and Sing-Yao Wu and Hyunwoo Oh and Chaewon Nam and Suyeon Koo and Mohsen Imani and Elaheh Bozorgzadeh and Nikil Dutt},
  title = {{AVERY: Intent-Driven Adaptive VLM Split Computing via Embodied Self-Awareness for Efficient Disaster Response Systems}},
  year = {2025},
  eprint = {2511.18151},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.18151},
}
Figure