Intelligent metasurface system for automatic tracking of moving targets and wireless communications based on computer vision

Weihan Li, Ma Qian, Yunfeng Zhang, Xianning Wu, Wei Wang, Shizhao Gao, Tianshuo Qiu, Tonghao Liu, Qiang Xiao, Jiaxuan Wei, Tingting Gu, Zhize Zhou, Fashuai Li, Qiang Cheng, Lianlin Li, Wenxuan Tang, Tie Jun Cui

TL;DR

Researchers have developed an autonomous, "closed-loop" wireless system that combines computer vision with digital programmable metasurfaces (DPMs) to track moving targets in real-time. By pairing a YOLOv4-tiny object detection network with a pre-trained artificial neural network (ANN) for beam steering, the system can automatically detect a target’s location and maintain a high-quality wireless link without any human intervention.

What Problem Does This Paper Address?

As we move deeper into the 5G and 6G eras, wireless networks face a massive challenge: maintaining high-speed connections with mobile users and devices (the "Internet of Things"). Traditional methods for tracking moving targets often rely on radar systems, which are frequently bulky, expensive, and computationally intensive.

Digital Programmable Metasurfaces (DPMs) have emerged as a lightweight, low-cost alternative for controlling electromagnetic (EM) waves. However, most existing metasurfaces are "passive" or "semi-manual"—they require a human operator or pre-defined schedules to change their reflection patterns. This paper tackles the gap between perception and action, creating a system that can "see" its environment and adapt its EM response dynamically.

Methods

The authors built an integrated hardware-software pipeline consisting of four main components:

  1. The "Eyes": An Intel RealSense RGB-D camera (RS-Camera) mounted on the metasurface to capture 40 frames per second of the scene.
  2. The "Perception": A YOLOv4-tiny convolutional neural network (CNN) processes the video feed to identify moving targets (in this case, model cars) and calculate their azimuth and elevation angles.
  3. The "Brain": A custom, pre-trained Artificial Neural Network (ANN). Instead of using slow, traditional numerical simulations to calculate how the metasurface should reflect waves, this ANN instantly maps the target's coordinates to the specific "coding sequences" (the 0s and 1s that control the surface elements).
  4. The "Actuator": A dual-polarized DPM made of 324 (18×18) elements. Each element contains PIN diodes that can be toggled by an FPGA (Field Programmable Gate Array) to shift the phase of reflected waves, effectively "steering" the microwave beam toward the target.

Datasets

The paper does not rely on a standard external dataset for its core contribution. Instead, the authors generated their own data to train the system:

  • Object Detection: The YOLOv4-tiny network was likely initialized with standard weights but optimized to detect the specific model cars used in the experimental setup.
  • Beam Steering ANN: The authors trained a fully connected neural network using a dataset of spatial angles (theta and phi) mapped to N-dimensional signal sequences. This allowed the system to bypass the "time-consuming" numerical simulations usually required for far-field EM control.

Key Insights and Ideas

  • Closing the Loop: The most significant insight is the integration of computer vision directly into the EM control loop. By treating the wireless environment as a visual scene, the system achieves "unsupervised" adaptation.
  • ANN as a Shortcut: Using an ANN to predict metasurface coding patterns is a clever engineering move. Traditional EM simulation is too slow for real-time tracking; the ANN provides the "instant" response needed for a target moving at 35 frames per second.
  • Dual-Polarization Flexibility: By designing elements that can control both x- and y-polarized waves independently, the system is more robust and capable of handling complex wireless communication schemes like frequency-division multiplexing.

Key Results

The system was validated through three primary experimental scenarios at a central frequency of 5.8 GHz:

  • Dynamic Tracking: The system successfully tracked a model car moving across a field of view, maintaining a directive beam with a scanning range of -40° to 40°.
  • RF Signal Strength: In tests using a portable RF detector, the system demonstrated that it could maintain peak power at the target's location, showing a clear "hump" in received energy as the car moved through the beam's path.
  • Real-Time Video Transmission: The authors successfully transmitted a live video feed to a moving car. The video remained smooth and stable while the car was in motion, proving that the beam-steering was accurate enough to support high-bandwidth data transfer.
  • Speed: The system achieved a sampling speed of approximately 0.2 seconds per refresh, sufficient for tracking typical indoor or slow-moving outdoor targets.

Limitations and Future Work

While impressive, the authors note a few areas for improvement:

  • Aperture Size: The current metasurface is relatively small (18×18 elements), which leads to the presence of "sidelobes" (wasted energy sent in unintended directions). A larger surface would provide a narrower, more efficient beam.
  • Latency: The 0.2s refresh rate is fast but might struggle with very high-speed targets (like a fast-moving vehicle on a highway).
  • Complexity of Environment: The experiments were conducted in controlled indoor environments. Future work will need to address how the system handles "clutter"—multiple moving objects, occlusions (objects blocking the view), and multi-path interference in complex urban settings.

Reference: Li, W., Ma, Q., Cui, T. J., et al. (2022). Intelligent metasurface system for automatic tracking of moving targets and wireless communications based on computer vision. Published in Nature Communications (2023). DOI: 10.1038/s41467-023-36645-3

Summary updated

Abstract

<title>Abstract</title> The fifth-generation (5G) wireless communication has an urgent need for target tracking. Digital programmable metasurface (DPM) may offer an intelligent and efficient solution owing to its powerful and flexible controls of electromagnetic waves and advantages of lower cost, less complexity and smaller size than the traditional antenna array. Here, we report an intelligent metasurface system to perform target tracking and wireless communications, in which computer vision integrated with a convolutional neural network (CNN) is used to automatically detect the locations of moving targets, and the dual-polarized DPM integrated with a pre-trained artificial neural network (ANN) serves to realize smart beam tracking and wireless communications. Three groups of experiments are conducted for demonstrating the intelligent system: detection of moving targets, detection of radio-frequency signals, and real-time wireless communications. The proposed method sets the stage for an integrated implementation of target identification, radio environment tracking, and wireless communications. This strategy opens up a new avenue for intelligent wireless networks and self-adaptive systems.
BibTeX
@inproceedings{li2023intelligent,
  author = {Weihan Li and Ma Qian and Yunfeng Zhang and Xianning Wu and Wei Wang and Shizhao Gao and Tianshuo Qiu and Tonghao Liu and Qiang Xiao and Jiaxuan Wei and Tingting Gu and Zhize Zhou and Fashuai Li and Qiang Cheng and Lianlin Li and Wenxuan Tang and Tie Jun Cui},
  title = {{Intelligent metasurface system for automatic tracking of moving targets and wireless communications based on computer vision}},
  booktitle = {Nature Communications},
  year = {2023},
  doi = {10.1038/s41467-023-36645-3},
}
Figure