Intelligent metasurface system for automatic tracking of moving targets and wireless communications based on computer vision

Weihan Li, Qian Ma, Che Liu, Yunfeng Zhang, Xianning Wu, Jiawei Wang, Shizhao Gao, Tianshuo Qiu, Tonghao Liu, Qiang Xiao, Jiaxuan Wei, Tingting Gu, Zhize Zhou, Fashuai Li, Qiang Cheng, Lianlin Li, Wenxuan Tang, Tie Jun Cui

Abstract

The fifth-generation (5G) wireless communication has an urgent need for target tracking. Digital programmable metasurface (DPM) may offer an intelligent and efficient solution owing to its powerful and flexible controls of electromagnetic waves and advantages of lower cost, less complexity and smaller size than the traditional antenna array. Here, we report an intelligent metasurface system to perform target tracking and wireless communications, in which computer vision integrated with a convolutional neural network (CNN) is used to automatically detect the locations of moving targets, and the dual-polarized DPM integrated with a pre-trained artificial neural network (ANN) serves to realize the smart beam tracking and wireless communications.

Introduction

As we move deeper into the 5G and 6G eras, wireless networks must support an explosion of mobile devices requiring ultra-reliable, high-speed links. Traditional "omnidirectional" broadcasting is inefficient; instead, beamforming is used to point energy directly at a user. However, tracking a moving user is difficult. Conventional methods often rely on bulky radar systems or complex feedback loops that are computationally expensive, slow, and power-hungry. There is a significant need for a hardware-software architecture that is low-cost, compact, and capable of "self-adaptive" tracking in dynamic environments.

Methods and System Architecture

The proposed system fuses computer vision with electromagnetic (EM) control through a closed-loop architecture consisting of three main pillars:

  1. The "Eyes" (Computer Vision): An RGB-D camera captures the scene at 40 frames per second. A lightweight YOLOv4-tiny neural network processes these images to identify the target and calculate its precise coordinates relative to the transmitter.
  2. The "Brain" (ANN-based Control): Instead of using slow numerical simulations, the team used a pre-trained Artificial Neural Network (ANN) based on a ResNet34 backbone. This ANN acts as an instantaneous "lookup table," taking the target's coordinates and outputting the optimal "coding sequence" for the metasurface in milliseconds.
  3. The "Actuator" (Digital Programmable Metasurface): The hardware is a 1-bit dual-polarized DPM consisting of an 18×18 array of elements. Each element contains PIN diodes that can be toggled via an FPGA to shift the phase of reflected EM waves, effectively "bending" the beam toward the target.
System architecture overview showing the integration of computer vision (YOLOv4-tiny), neural networks (ResNet34), and the digital programmable metasurface for target tracking.

Metasurface Unit Cell Design

The metasurface is composed of unit cells capable of dual-polarization control. By switching the states of integrated PIN diodes, the system can independently control the reflection characteristics for different polarizations, enhancing robustness in real-world environments where signals may shift polarization during propagation.

Detailed view of the metasurface unit cell design (a), its four switching states (c), and the corresponding reflection amplitude performance (d).

Key Insights

  • Cross-Modal Fusion: Using the visual spectrum (light) to solve a problem in the microwave spectrum allows the system to bypass the need for constant feedback signals from the user.
  • Eliminating the Simulation Bottleneck: Moving beamforming logic into a pre-trained ANN allows the system to react to movement almost instantly, avoiding heavy numerical optimization.
  • Dual-Polarization Versatility: Independent control of two orthogonal polarizations makes the system more robust for real-world wireless signals that reflect off complex terrain.

Experimental Results

The system was validated through three experimental stages:

  • Tracking Accuracy: The system successfully tracked a model car, refreshing the beam direction every 0.2 seconds.
  • Beam Precision: The DPM could steer beams from -40° to 40° with an angular precision of approximately 1°.
  • Real-Time Communication: In a live test, the system transmitted a high-definition video stream to a moving receiver at 170 Mbps using QPSK modulation. The Bit Error Rate (BER) remained stable at $10^{-5}$, proving accurate tracking during movement.

Conclusion

The proposed method sets the stage for an integrated implementation of target identification, radio environment tracking, and wireless communications. While currently limited by line-of-sight dependency and single-user scaling, this strategy opens up an avenue for intelligent wireless networks and self-adaptive systems in the future.

Summary updated

Abstract

The fifth-generation (5G) wireless communication has an urgent need for target tracking. Digital programmable metasurface (DPM) may offer an intelligent and efficient solution owing to its powerful and flexible controls of electromagnetic waves and advantages of lower cost, less complexity and smaller size than the traditional antenna array. Here, we report an intelligent metasurface system to perform target tracking and wireless communications, in which computer vision integrated with a convolutional neural network (CNN) is used to automatically detect the locations of moving targets, and the dual-polarized DPM integrated with a pre-trained artificial neural network (ANN) serves to realize the smart beam tracking and wireless communications. Three groups of experiments are conducted for demonstrating the intelligent system: detection and identification of moving targets, detection of radio-frequency signals, and real-time wireless communications. The proposed method sets the stage for an integrated implementation of target identification, radio environment tracking, and wireless communications. This strategy opens up an avenue for intelligent wireless networks and self-adaptive systems.
BibTeX
@inproceedings{li2023intelligent,
  author = {Weihan Li and Qian Ma and Che Liu and Yunfeng Zhang and Xianning Wu and Jiawei Wang and Shizhao Gao and Tianshuo Qiu and Tonghao Liu and Qiang Xiao and Jiaxuan Wei and Tingting Gu and Zhize Zhou and Fashuai Li and Qiang Cheng and Lianlin Li and Wenxuan Tang and Tie Jun Cui},
  title = {{Intelligent metasurface system for automatic tracking of moving targets and wireless communications based on computer vision}},
  booktitle = {Nature Communications},
  year = {2023},
  doi = {10.1038/s41467-023-36645-3},
}
Figure