Abstract
There is a growing literature demonstrating the feasibility of using Radio Frequency (RF) signals to enable key computer vision tasks in the presence of occlusions and poor lighting. It leverages that RF signals traverse walls and occlusions to deliver through-wall pose estimation, action recognition, scene captioning, and human re-identification. However, unlike RGB datasets which can be labeled by human workers, labeling RF signals is a daunting task because such signals are not human interpretable. Yet, it is fairly easy to collect unlabelled RF signals. It would be highly beneficial to use such unlabeled RF data to learn useful representations in an unsupervised manner. Thus, in this paper, we explore the feasibility of adapting RGB-based unsupervised representation learning to RF signals. We show that while contrastive learning has emerged as the main technique for unsupervised representation learning from images and videos, such methods produce poor performance when applied to sensing humans using RF signals. In contrast, predictive unsupervised learning methods learn high-quality representations that can be used for multiple downstream RF-based sensing tasks. Our empirical results show that this approach outperforms state-of-the-art RF-based human sensing on various tasks, opening the possibility of unsupervised representation learning from this novel modality.
Introduction
Radio Frequency (RF) sensing has emerged as a powerful alternative to traditional computer vision, particularly in scenarios requiring privacy or visibility through occlusions like walls and smoke. While deep learning has revolutionized RF-based tasks such as 3D pose estimation and action recognition, these models typically rely on massive labeled datasets. Unlike images, where a human can easily identify a "person" or an "action," raw RF data consists of abstract heatmaps that are unintelligible to the human eye.
Currently, labeling RF data requires expensive, synchronized multi-modal setups (e.g., using cameras to generate ground-truth labels for the RF sensor). This dependency limits the scalability of RF sensing and prevents its deployment in private environments where cameras are prohibited. To overcome this, we investigate unsupervised representation learning—specifically, how to enable models to learn the "semantics" of human movement from raw, unlabeled radio waves.
The TGUL Framework
We propose Trajectory-Guided Unsupervised Learning (TGUL), a framework designed to bridge the gap between raw physical signals and deep learning. The primary challenge in unsupervised RF learning is that the human signal often occupies less than 1% of the data, with the rest being background noise or reflections from furniture. Standard unsupervised techniques like contrastive learning (e.g., SimCLR) often fail because the model learns "shortcuts"—easy features like the person's distance from the sensor—rather than the complex geometry of human pose.
TGUL operates in two distinct stages:
- Signal Processing for ROI Detection: Instead of relying on a neural network to find the person, we use a classical radar technique called WiTrack. By performing background subtraction and tracking moving reflections, we can automatically generate a "bounding box" or Region of Interest (ROI) around a person's trajectory through a room.
- Predictive Representation Learning: Once the ROI is identified, we "zoom in" on the human signal. We found that predictive learning (inpainting) is significantly more effective than contrastive learning for this modality. By masking temporal segments of the RF signal and requiring the model to reconstruct the missing frames, the network is forced to learn the spatio-temporal dynamics of how a human body moves.
Experimental Results
We evaluated TGUL on two large-scale real-world datasets: RF-MMD (35 actions, 30 volunteers) and RRD (person re-identification across homes and campuses).
- 3D Pose Estimation: TGUL with inpainting reduced pose estimation error by 15% in fixed-feature settings compared to standard baselines.
- Action Recognition: The framework achieved a 19.5% relative improvement in mean Average Precision (mAP).
- Person Re-Identification: On the RRD-Campus dataset, TGUL delivered a massive 77.2% relative improvement in mAP, demonstrating its ability to learn identity-specific features without labels.
- Data Efficiency: In low-label regimes, TGUL improved performance by 11.3% by effectively mining information from additional unlabeled data.
Conclusion
TGUL demonstrates that unsupervised learning is not only possible but highly effective for RF-based human sensing when guided by physical trajectories. By moving away from contrastive "shortcuts" and toward predictive tasks like inpainting, we can leverage the vast amounts of unlabeled RF data available in the real world. Future work will focus on extending this framework to multi-person environments and improving sensitivity to static human subjects.
Summary updated