Hierarchical Reinforcement Learning for Cooperative Air-Ground Delivery in Urban System

Songxin Lei, Chunming Ma, Haomin Wen, Yexin Li, Lizhenghe Chen, Qianyu Yang, Fugee Tsung, Lei Chen, Sijie Ruan, Yuxuan Liang

Abstract

Cooperative air-ground delivery has emerged as a promising logistics paradigm by leveraging the complementary strengths of UAVs and ground carriers. However, effective dispatching in such heterogeneous systems faces two critical challenges: i) the heterogeneity between flight and road dynamics, ii) the scalability bottleneck raised by the exponential decision variables in large-scale fleets. To address these challenges, we propose HRL4AG, a Hierarchical Reinforcement Learning framework for cooperative Air-Ground delivery. Specifically, HRL4AG employs a high-level manager to tackle the scalability bottleneck by decomposing the joint action space, and mode-specific workers that encode distinct flight and road dynamics to address the heterogeneity. Furthermore, a novel internal reward mechanism is designed to guide the hierarchical policy learning, addressing the credit assignment problem in sparse-reward settings. Extensive experiments on two real-world datasets and an evaluation platform demonstrate that HRL4AG significantly outperforms state-of-the-art baselines, improving the delivery success rate by up to 26% while achieving an 80-fold increase in computational efficiency.

The Challenge of Hybrid Logistics

The rapid growth of on-demand logistics has pushed the industry toward cooperative air-ground delivery. Unmanned Aerial Vehicles (UAVs) offer high-speed, traffic-free transport but are limited by battery life and payload capacity. Conversely, ground carriers possess high endurance and capacity but are subject to road congestion.

While combining these modes is theoretically efficient, real-time dispatching for a mixed fleet presents a massive sequential decision-making challenge. Existing solutions struggle with two primary limitations:

  1. The Heterogeneity Gap: Most Reinforcement Learning (RL) models treat all vehicles as homogeneous. However, drones operate in 3D space with Euclidean distances and strict battery limits, while carriers navigate 2D road networks with traffic constraints.
  2. The Scalability Bottleneck: As fleet sizes grow, the number of possible order-to-vehicle assignments explodes exponentially. Traditional centralized RL models often fail to handle this massive joint action space, leading to slow training and high inference latency.
Challenges in air-ground delivery: heterogeneity and scalability.

The HRL4AG Framework

To overcome these hurdles, the HRL4AG (Hierarchical Reinforcement Learning for cooperative Air-Ground delivery) framework adopts a "divide and conquer" strategy, breaking the dispatch problem into a two-tier hierarchy:

  • Heterogeneous State Encoding: The model utilizes type-specific Transformer encoders to process UAVs, ground carriers, and orders separately. This captures unique physical constraints while maintaining permutation invariance.
  • High-Level Manager Agent: This agent manages the macroscopic environment by making a binary decision for each order: Air fleet or Ground fleet. This high-level selection instantly halves the action space for the lower level.
  • Low-Level Worker Agents: Conditioned on the Manager's decision, mode-specific worker agents (one for UAVs, one for carriers) evaluate vehicle feasibility and generate matching scores to finalize assignments.
  • Trajectory-Driven Reward Shaping: To address the sparse reward problem (where success only occurs at the end of a delivery), the framework provides dense feedback for intermediate milestones like matching and picking up. An intrinsic reward mechanism ensures workers align their assignments with the Manager's high-level goals.
The HRL4AG architecture featuring hierarchical decision-making and heterogeneous state encoding.

Key Insights and Performance

By delegating mode selection to a high-level manager, HRL4AG bypasses the curse of dimensionality. This architectural choice allows the model to achieve sub-second inference times, making it viable for real-world deployment. Furthermore, by treating heterogeneity as a feature—using distinct worker agents for different vehicle types—the system naturally learns to exploit physical differences, such as assigning time-critical requests to UAVs and heavy-traffic tasks to carriers.

Experimental Results

The framework was evaluated on real-world datasets from Shanghai (~20,000 orders) and Chengdu (~96,000 orders). Key findings include:

  • Superior Delivery Performance: HRL4AG improved delivery success rates by 18–28% on the Shanghai dataset and 20–25% on the Chengdu dataset compared to state-of-the-art baselines.
  • Massive Computational Speedup: The hierarchical design yielded an 80-fold increase in efficiency. While flat RL models took up to 37 seconds per decision epoch, HRL4AG completed them in 0.17 to 0.83 seconds.
  • Ablation Validation: Removing the hierarchical structure caused inference time to skyrocket from 0.35s to over 22s, proving the necessity of the bi-level approach.

Conclusion

HRL4AG demonstrates that hierarchical reinforcement learning can effectively bridge the gap between complex, heterogeneous vehicle dynamics and the need for scalable, real-time logistics dispatching. Future work will focus on incorporating stochastic environmental variables, such as shifting weather and uncertain energy consumption, to further enhance real-world robustness.

Summary updated

Abstract

Cooperative air-ground delivery has emerged as a promising logistics paradigm by leveraging the complementary strengths of UAVs and ground carriers. However, effective dispatching in such heterogeneous systems faces two critical challenges: i) the heterogeneity between flight and road dynamics, ii) the scalability bottleneck raised by the exponential decision variables in large-scale fleets. To address these challenges, we propose HRL4AG, a Hierarchical Reinforcement Learning framework for cooperative Air-Ground delivery. Specifically, HRL4AG employs a high-level manager to tackle the scalability bottleneck by decomposing the joint action space, and mode-specific workers that encode distinct flight and road dynamics to address the heterogeneity. Furthermore, a novel internal reward mechanism is designed to guide the hierarchical policy learning, addressing the credit assignment problem in sparse-reward settings. Extensive experiments on two real-world datasets and an evaluation platform demonstrate that HRL4AG significantly outperforms state-of-the-art baselines, improving the delivery success rate by up to 26% while achieving an 80-fold increase in computational efficiency.
BibTeX
@misc{lei2026hierarchical,
  author = {Songxin Lei and Chunming Ma and Haomin Wen and Yexin Li and Lizhenghe Chen and Qianyu Yang and Fugee Tsung and Lei Chen and Sijie Ruan and Yuxuan Liang},
  title = {{Hierarchical Reinforcement Learning for Cooperative Air-Ground Delivery in Urban System}},
  year = {2026},
  eprint = {2602.12913},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.12913},
}
Figure