Age of Information Minimization in Vehicular Edge Computing Networks: A Mask-Assisted Hybrid PPO-Based Method

Xiaoli Qin, Zhifei Zhang, Chanyuan Meng, Rui Dong, Ke Xiong, Pingyi FanNetwork 2025 · DOI

TL;DR

This paper tackles the challenge of minimizing the "Age of Information" (AoI) in vehicular edge computing (VEC) networks, where timely data is crucial for intelligent applications like autonomous driving. They propose a novel Mask-Assisted Hybrid Proximal Policy Optimization (MHPPO) method that effectively balances task offloading and resource allocation, outperforming existing methods by significantly reducing AoI.

What Problem Does This Paper Address?

The rapid growth of connected vehicles and data-intensive applications in the Internet of Vehicles (IoV) has created a massive demand for computational resources and low latency. Vehicular Edge Computing (VEC) offers a solution by offloading tasks to roadside units (RSUs) or other vehicles. However, existing VEC research often focuses on minimizing latency or energy consumption, which doesn't fully capture the "freshness" of information. The "Age of Information" (AoI) is a more comprehensive metric that measures the time elapsed since the latest information was generated. This paper addresses the problem of minimizing AoI in VEC networks by jointly optimizing task offloading (to RSUs or other vehicles) and subcarrier allocation, a complex problem due to time-varying channels and coupled continuous/discrete variables.

Methods

The core of this paper is the proposed Mask-Assisted Hybrid Proximal Policy Optimization (MHPPO) method. This approach builds upon the Proximal Policy Optimization (PPO) algorithm, a popular deep reinforcement learning (DRL) technique.

Here's a breakdown of the key methodological contributions:

  • Hybrid Actor-Critic Network: MHPPO utilizes a hybrid actor-critic architecture. The actor network is responsible for generating actions (decisions on task offloading and resource allocation), while the critic network evaluates the quality of these actions.
  • Mixed Action Space: The problem involves both continuous variables (e.g., offloading ratio) and discrete variables (e.g., selecting an offloading destination, subcarrier allocation). MHPPO is designed to handle this "mixed action space" by having separate components within the actor network for continuous and discrete actions.
  • Action Masking Mechanism: A crucial innovation is the action masking mechanism. This mechanism intelligently filters out invalid actions that might arise due to physical constraints, such as the limited communication range between vehicles. By masking these invalid actions, the agent is prevented from selecting them, leading to more efficient learning and preventing suboptimal decisions.
  • PPO-Clip for Policy Updates: The paper employs the PPO-Clip approach for updating the actor and critic networks. This method uses a specialized objective function to limit the changes between old and new policies, promoting stable and efficient learning.

Datasets

Not discussed in the paper. The paper focuses on simulation-based evaluation using a defined system model.

Key Insights and Ideas

  • AoI is a Superior Metric for Timeliness: The paper strongly advocates for AoI as a more appropriate metric than traditional latency for evaluating information freshness in time-sensitive VEC applications.
  • Action Masking is Crucial for Real-World Constraints: The introduction of action masking is a significant practical improvement. It directly addresses the limitations imposed by physical constraints like communication range, which are often overlooked in purely theoretical models. This makes the learned policy more robust and applicable to real-world scenarios.
  • Hybrid DRL for Mixed Action Spaces: The successful application of a hybrid DRL approach to handle mixed continuous and discrete action spaces in VEC is a key takeaway. This demonstrates the power of DRL in tackling complex optimization problems with diverse variable types.
  • V2V Offloading is Essential for Resource Utilization: By considering both V2I and V2V offloading, the method effectively leverages underutilized computational resources in neighboring vehicles, leading to better overall system performance.

Key Results

The proposed MHPPO method demonstrated significant improvements in reducing the average Age of Information (AoI):

  • 28.9% reduction in AoI compared to the standard Hybrid PPO (HPPO) method.
  • 23% reduction in AoI compared to the Mask-Assisted Deep Deterministic Policy Gradient (MDDPG) method.
  • 38.2% reduction in AoI compared to the conventional Deep Deterministic Policy Gradient (DDPG) method.
  • The method also showed substantial improvements over random action selection (RAMDOM) and local computation (LOC) strategies.
  • The paper also shows that MHPPO is robust to variations in the number of task vehicles, data size, and service vehicles, maintaining superior performance across different scenarios.

Limitations and Future Work

  • Simulation-Based: The results are based on simulations, and real-world deployment might introduce additional complexities.
  • Simplified Traffic Model: The paper considers vehicles moving in the same direction and at constant speeds. Future work could explore more complex traffic dynamics, including bidirectional traffic and varying speeds.
  • Future Work: The authors plan to expand their research to address more complex scenarios, including:
  • Bidirectional vehicle mobility and multi-RSU environments.
  • Investigating incentive mechanisms for service vehicles to encourage active participation.
  • Exploring more complex task arrival models and scheduling strategies.

Summary updated

BibTeX
@article{qin2025age,
  author = {Xiaoli Qin and Zhifei Zhang and Chanyuan Meng and Rui Dong and Ke Xiong and Pingyi Fan},
  title = {{Age of Information Minimization in Vehicular Edge Computing Networks: A Mask-Assisted Hybrid PPO-Based Method}},
  journal = {Network},
  year = {2025},
  doi = {10.3390/network5020012},
}
Figure