Unsupervised Learning for Human Sensing Using Radio Signals

Tianhong Li, Lijie Fan, Yuan Yuan, Dina Katabi

Radio frequency (RF) signals, such as WiFi, offer a unique modality for human sensing. Unlike traditional cameras, RF-based vision can operate in total darkness, see through walls, and preserve user privacy by avoiding the capture of recognizable facial features. However, the development of these systems has been historically hindered by a massive data labeling bottleneck. This paper, presented at WACV 2022 by researchers from MIT CSAIL, introduces Trajectory-Guided Unsupervised Learning (TGUL), a framework designed to unlock the potential of unlabeled RF data.

The Labeling Bottleneck in RF Sensing

While humans can easily label objects in images, RF signals are unintelligible to the human eye. Traditionally, labeling RF datasets requires a synchronized camera system to provide "ground truth" labels. This process is expensive, difficult to calibrate, and often impossible to implement in the very environments where RF sensing is most valuable, such as private homes or low-light scenarios.

RF signals (top) vs. corresponding RGB camera footage (bottom). The heatmaps illustrate how human movement appears as sparse reflections in the radio spectrum.

Furthermore, standard unsupervised techniques like contrastive learning, which work well for images, often fail with RF data. RF signals are extremely sparse—with the human subject typically occupying less than 1% of the data—and models frequently learn "shortcuts" by focusing on static background reflections rather than the actual human features.

Trajectory-Guided Unsupervised Learning (TGUL)

To overcome these challenges, the authors proposed TGUL, a two-step framework that leverages signal processing to guide neural representation learning:

  1. Signal Processing for ROI Detection: The system uses a traditional signal processing technique (WiTrack) to detect moving objects. By identifying the trajectory of a person, the model can "zoom in" on a specific 1m x 1.5m Region of Interest (ROI). This effectively crops out the 99% of the signal that constitutes background noise from walls and furniture.
  2. Predictive Unsupervised Learning (Inpainting): Instead of comparing different views of the same data, TGUL employs a predictive task. The model randomly masks out segments of the RF signal and is tasked with reconstructing the missing parts. To succeed at this "inpainting" task, the model must develop an internal understanding of human kinematics and how bodies reflect radio waves over time.

Experimental Evaluation

The researchers evaluated TGUL on two major real-world datasets: RF-MMD (25 hours of data for pose estimation and action recognition) and RRD (a person re-identification dataset covering campus and home environments).

Key Results

TGUL consistently outperformed both random initialization and supervised baselines across multiple tasks:

  • 3D Pose Estimation: Pre-training with TGUL on unlabeled data reduced the error rate by 11.3% compared to standard supervised methods.
  • Action Recognition: The framework achieved a 19.5% relative improvement in mean Average Precision (mAP).
  • Person Re-ID: The most significant gains were seen in person re-identification, with a 77.2% relative improvement in mAP on campus data and 27.9% in home environments.

Insights and Future Directions

A critical insight from this work is that predictive learning tasks are more robust for RF signals than contrastive ones. By forcing the model to fill in missing temporal or spatial data, it learns generalizable features of human movement rather than environmental shortcuts.

While the current framework relies on movement for initial ROI detection—meaning it may struggle with subjects who remain perfectly still—it represents a major step forward. By reducing the dependency on expensive labeled data, TGUL paves the way for deploying sophisticated human sensing systems in diverse, real-world environments where traditional cameras cannot go.

Summary updated

Abstract

There is a growing literature demonstrating the feasibility of using Radio Frequency (RF) signals to enable key computer vision tasks in the presence of occlusions and poor lighting. It leverages that RF signals traverse walls and occlusions to deliver through-wall pose estimation, action recognition, scene captioning, and human re-identification. However, unlike RGB datasets which can be labeled by human workers, labeling RF signals is a daunting task because such signals are not human interpretable. Yet, it is fairly easy to collect unlabelled RF signals. It would be highly beneficial to use such unlabeled RF data to learn useful representations in an unsupervised manner. Thus, in this paper, we explore the feasibility of adapting RGB-based unsupervised representation learning to RF signals. We show that while contrastive learning has emerged as the main technique for unsupervised representation learning from images and videos, such methods produce poor performance when applied to sensing humans using RF signals. In contrast, predictive unsupervised learning methods learn high-quality representations that can be used for multiple downstream RF-based sensing tasks. Our empirical results show that this approach outperforms state-of-the-art RF-based human sensing on various tasks, opening the possibility of unsupervised representation learning from this novel modality.
BibTeX
@inproceedings{li2022unsupervised,
  author = {Tianhong Li and Lijie Fan and Yuan Yuan and Dina Katabi},
  title = {{Unsupervised Learning for Human Sensing Using Radio Signals}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
  doi = {10.1109/wacv51458.2022.00116},
}
Figure