Efficient Onboard Vision-Language Inference in UAV-Enabled Low-Altitude Economy Networks via LLM-Enhanced Optimization

Yang Li, Ruichen Zhang, Yinqiu Liu, Guangyuan Liu, Dusit Niyato, Abbas Jamalipour, Xianbin Wang, Dong In Kim

Abstract

The rapid advancement of Low-Altitude Economy Networks (LAENets) has enabled a variety of applications, including aerial surveillance, environmental sensing, and semantic data collection. To support these scenarios, unmanned aerial vehicles (UAVs) equipped with onboard vision-language models (VLMs) offer a promising solution for real-time multimodal inference. However, ensuring both inference accuracy and communication efficiency remains a significant challenge due to limited onboard resources and dynamic network conditions. In this paper, we first propose a UAV-enabled LAENet system model that jointly captures UAV mobility, user-UAV communication, and the onboard visual question answering (VQA) pipeline. Based on this model, we formulate a mixed-integer non-convex optimization problem to minimize task latency and power consumption under user-specific accuracy constraints. To solve the problem, we design a hierarchical optimization framework composed of two parts: (i) an Alternating Resolution and Power Optimization (ARPO) algorithm for resource allocation under accuracy constraints, and (ii) a Large Language Model-augmented Reinforcement Learning Approach (LLaRA) for adaptive UAV trajectory optimization. The large language model (LLM) serves as an expert in refining reward design of reinforcement learning in an offline fashion, introducing no additional latency in real-time decision-making. Numerical results demonstrate the efficacy of our proposed framework in improving inference performance and communication efficiency under dynamic LAENet conditions.

Introduction

As we transition toward a robust "Low-Altitude Economy," Unmanned Aerial Vehicles (UAVs) are evolving from simple remote-controlled aircraft into intelligent agents capable of complex perception and reasoning. Integrating Vision-Language Models (VLMs) directly onto UAVs enables sophisticated tasks such as Visual Question Answering (VQA). For instance, a drone could autonomously identify specific objects or anomalies in a crowded urban environment based on natural language queries from ground users.

However, deploying VLMs on resource-constrained UAVs presents a severe "accuracy-efficiency" trade-off. High-resolution imagery is essential for accurate inference but imposes significant transmission delays and computational burdens. Furthermore, the UAV's physical trajectory dynamically alters signal strength and energy consumption. Addressing these challenges requires a holistic optimization of communication, computation, and flight paths—a problem that is mathematically NP-hard and highly non-convex.

System architecture of the UAV-assisted VLM inference service within a LAENet environment.

Methodology

To tackle the joint optimization of resource allocation and trajectory control, we propose a hierarchical framework that decomposes the original complex problem into manageable sub-problems.

Hierarchical Optimization Framework

The core of our approach lies in separating "fast" variables (transmission power and image resolution) from "slow" variables (UAV trajectory). As illustrated in the framework overview, the original optimization problem $\mathbb{P}_0$ is decomposed into resource allocation ($\mathbb{P}_1$) and trajectory control ($\mathbb{P}_3$).

The hierarchical optimization framework showing problem decomposition into ARPO for resource allocation and LLaRA for trajectory control.

ARPO: Alternating Resolution and Power Optimization

The ARPO algorithm handles the immediate resource allocation for each user. It utilizes a Branch and Bound (B&B) algorithm to select optimal discrete image resolutions based on empirical lookup tables that map resolution to expected accuracy and inference latency. Once resolutions are fixed, the algorithm applies Karush-Kuhn-Tucker (KKT) conditions to optimize user transmit power, ensuring efficient uplink communication.

LLaRA: LLM-augmented Reinforcement Learning Approach

Trajectory planning is a long-horizon problem addressed through Deep Reinforcement Learning (DRL). A novel contribution of this work is the use of a Large Language Model (LLM), specifically GPT-4o, as an offline reward designer.

Instead of manual reward engineering, the LLM acts as an expert that receives the system model and iteratively refines Python code for the DRL reward function based on training performance. This "LLM-in-the-loop" approach allows for sophisticated reward structures, such as "tail-aware" rewards that prioritize users with the longest wait times, without adding any computational latency during real-time flight operations.

Key Insights and Results

Our evaluation using benchmarks like TextVQA and empirical profiling of models such as LLaVA-1.5-13B and LLaVA-HR-7B yielded several critical insights:

  • Resolution vs. Model Size: We found that optimizing image resolution is often more impactful than increasing model parameters. A smaller, high-resolution model (e.g., LLaVA-HR-7B) frequently outperforms much larger models (e.g., Qwen-225B) when the latter is restricted to low-resolution inputs due to bandwidth constraints.
  • Latency Reduction: The ARPO-LLaRA framework achieved a 45% reduction in task latency compared to random policies and a 25% improvement over standard geometric heuristics.
  • LLM-Enhanced RL Performance: The LLM-designed reward function provided a 13.7% improvement in latency reduction over traditional DRL baselines with human-designed rewards, demonstrating the LLM's capability as a meta-optimizer.
  • Accuracy Maintenance: By intelligently balancing resolution and power, the system maintained inference accuracy above 67% while significantly optimizing the communication uplink.

Conclusion

This work demonstrates that hierarchical optimization, combined with the reasoning capabilities of LLMs for offline reward design, can effectively bridge the gap between complex VLM requirements and the limited resources of UAV-enabled networks. Future research will explore multi-UAV coordination and the impact of highly mobile ground users on system performance.

Summary updated

Abstract

The rapid advancement of Low-Altitude Economy Networks (LAENets) has enabled a variety of applications, including aerial surveillance, environmental sensing, and semantic data collection. To support these scenarios, unmanned aerial vehicles (UAVs) equipped with onboard vision-language models (VLMs) offer a promising solution for real-time multimodal inference. However, ensuring both inference accuracy and communication efficiency remains a significant challenge due to limited onboard resources and dynamic network conditions. In this paper, we first propose a UAV-enabled LAENet system model that jointly captures UAV mobility, user-UAV communication, and the onboard visual question answering (VQA) pipeline. Based on this model, we formulate a mixed-integer non-convex optimization problem to minimize task latency and power consumption under user-specific accuracy constraints. To solve the problem, we design a hierarchical optimization framework composed of two parts: (i) an Alternating Resolution and Power Optimization (ARPO) algorithm for resource allocation under accuracy constraints, and (ii) a Large Language Model-augmented Reinforcement Learning Approach (LLaRA) for adaptive UAV trajectory optimization. The large language model (LLM) serves as an expert in refining reward design of reinforcement learning in an offline fashion, introducing no additional latency in real-time decision-making. Numerical results demonstrate the efficacy of our proposed framework in improving inference performance and communication efficiency under dynamic LAENet conditions.
BibTeX
@misc{li2025efficient,
  author = {Yang Li and Ruichen Zhang and Yinqiu Liu and Guangyuan Liu and Dusit Niyato and Abbas Jamalipour and Xianbin Wang and Dong In Kim},
  title = {{Efficient Onboard Vision-Language Inference in UAV-Enabled Low-Altitude Economy Networks via LLM-Enhanced Optimization}},
  year = {2025},
  eprint = {2510.10028},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.10028},
}
Figure