MetaSketch: Wireless Semantic Segmentation by Metamaterial Surfaces

Jingzhi Hu, Hongliang Zhang, Kaigui Bian, Zhu Han, H. Vincent Poor, Lingyang SongarXiv 2021 · DOI · arXiv

Abstract

Semantic segmentation is a process of partitioning an image into multiple segments for recognizing humans and objects, which can be widely applied in scenarios such as healthcare and safety monitoring. To avoid privacy violation, using RF signals instead of an image for human and object recognition has gained increasing attention. However, human and object recognition by using RF signals is usually a passive signal collection and analysis process without changing the radio environment, and the recognition accuracy is restricted significantly by unwanted multi-path fading, and/or the limited number of independent channels between RF transceivers in uncontrollable radio environments. This paper introduces MetaSketch, a novel RF-sensing system that performs semantic recognition and segmentation for humans and objects by making the radio environment reconfigurable. A metamaterial surface is incorporated into MetaSketch and diversifies the information carried by RF signals. Using compressive sensing techniques, MetaSketch reconstructs a point cloud consisting of the reflection coefficients of humans and objects at different spatial points, and recognizes the semantic meaning of the points by using symmetric multilayer perceptron groups. Our evaluation results show that MetaSketch is capable of generating favorable radio environments and extracting exact point clouds, and labeling the semantic meaning of the points with an average error rate of less than 1% in an indoor space.

Introduction

Semantic segmentation—the process of partitioning an image and labeling every pixel (e.g., "this is a human," "this is a chair")—is a cornerstone of computer vision. However, using cameras for this in sensitive areas like hospitals or homes raises significant privacy concerns.

While researchers have turned to Radio Frequency (RF) signals (like WiFi or millimeter waves) as a "privacy-friendly" alternative, traditional RF sensing is usually passive. It relies on whatever signals happen to be in the room, which are often degraded by multi-path fading and a limited number of independent channels. This makes it incredibly difficult to reconstruct a detailed "image" or point cloud of a scene without the high-resolution optics of a camera.

System Architecture

MetaSketch shifts the paradigm from passive sensing to active environment reconfiguration. The system consists of three main modules:

1. Radio Environment Reconfiguration Module

The heart of the system is a metasurface (a Reconfigurable Intelligent Surface or RIS). This surface contains 2304 tiny, electrically controllable elements. Each element consists of metal patches connected by PIN diodes. By switching the states of these diodes, the system can "shape" how RF signals bounce around the room.

The MetaSketch metasurface architecture, showing the constituent elements (PIN diodes and metal patches) and the reflected signal beam.

The authors developed an optimization algorithm to find configuration matrices that minimize "mutual coherence," essentially ensuring that the signals reflected from different points in space are as distinct as possible.

2. Point Cloud Extraction Module

Using compressive sensing ($l_1$-norm minimization), MetaSketch reconstructs the reflection coefficients of 400 specific points in a 3D grid. Because most points in a room are empty space, the data is "sparse," allowing the system to reconstruct a detailed point cloud from a relatively small number of RF measurements.

3. Semantic Segmentation Module

To classify the reconstructed points, the authors used Symmetric Multi-Layer Perceptron (MLP) groups. Unlike standard neural networks that expect ordered pixels, these MLPs are designed to handle unordered sets of points (inspired by PointNet). The network extracts both local features (individual point characteristics) and global features (the overall shape of the scene) to label each point.

Experimental Setup and Datasets

Because this is a specialized hardware-software co-design, the authors built their own experimental setup in an anechoic chamber to validate the system.

Experimental setup showing the metasurface states and the resulting phase distributions in a lab environment.
  • Hardware: Two USRP (Universal Software Radio Peripheral) devices acting as transmitter and receiver, low-noise amplifiers, and a custom-built metasurface.
  • Environment: A $1.6\text{ m}^3$ indoor "target space" divided into 400 blocks ($0.4 \times 0.2 \times 0.2\text{ m}^3$ each).
  • Data: They generated a ground-truth dataset of 64 distinct point cloud scenarios involving a human, a suitcase, a laptop, and a bottle.

Key Results

  • High Accuracy: In an indoor space, MetaSketch achieved an average error rate of less than 1% when labeling points.
  • Optimization Gains: Using an optimized metasurface configuration (rather than a random one) significantly lowered the "loss" during training and improved the clarity of the reconstructed point clouds.
  • Configuration Efficiency: With just $K=10$ different metasurface configurations, the system could perform accurate segmentation. Increasing to $K=50$ reduced the error rate to near $0.1\%$, though at the cost of slower data collection.

Discussion and Future Work

While MetaSketch demonstrates the power of active RF sensing, it faces several challenges. The current resolution is limited to $0.4 \times 0.2 \times 0.2\text{ m}^3$ blocks; finer details would require smaller calibration patches and more measurements. Additionally, the data collection phase currently takes time (about 0.1s per configuration), meaning rapid movement can blur the point cloud. Finally, the system requires an initial manual calibration phase where a metal patch is placed in every grid block. Automating this calibration or using purely theoretical models is a key area for future research.

Summary updated

Abstract

Semantic segmentation is a process of partitioning an image into multiple segments for recognizing humans and objects, which can be widely applied in scenarios such as healthcare and safety monitoring. To avoid privacy violation, using RF signals instead of an image for human and object recognition has gained increasing attention. However, human and object recognition by using RF signals is usually a passive signal collection and analysis process without changing the radio environment, and the recognition accuracy is restricted significantly by unwanted multi-path fading, and/or the limited number of independent channels between RF transceivers in uncontrollable radio environments. This paper introduces MetaSketch, a novel RF-sensing system that performs semantic recognition and segmentation for humans and objects by making the radio environment reconfigurable. A metamaterial surface is incorporated into MetaSketch and diversifies the information carried by RF signals. Using compressive sensing techniques, MetaSketch reconstructs a point cloud consisting of the reflection coefficients of humans and objects at different spatial points, and recognizes the semantic meaning of the points by using symmetric multilayer perceptron groups. Our evaluation results show that MetaSketch is capable of generating favorable radio environments and extracting exact point clouds, and labeling the semantic meaning of the points with an average error rate of less than 1% in an indoor space.
BibTeX
@misc{hu2021metasketch,
  author = {Jingzhi Hu and Hongliang Zhang and Kaigui Bian and Zhu Han and H. Vincent Poor and Lingyang Song},
  title = {{MetaSketch: Wireless Semantic Segmentation by Metamaterial Surfaces}},
  year = {2021},
  eprint = {2108.06456},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2108.06456},
}
Figure