Neural Architectural Nonlinear Pre-Processing for mmWave Radar-based Human Gesture Perception

Hankyul Baek, Yoo Jeong, Ha, Minjae Yoo, Soyi Jung, Joongheon KimarXiv 2022 · arXiv

TL;DR

This paper proposes a two-stage deep learning pipeline to improve human gesture recognition using mmWave radar, a sensor that works in total darkness where cameras fail. By using a U-Net for nonlinear denoising followed by an EfficientNet for classification, the researchers achieved over 90% accuracy in identifying hand movements, even in noisy environments typical of autonomous driving.

What Problem Does This Paper Address?

As we move toward more sophisticated "on-driving" computing environments, cars need better ways to understand driver intent. While cameras are standard for gesture recognition, they struggle in low-light or variable-glare conditions. Millimeter-wave (mmWave) radar is an excellent alternative because it is low-cost, compact, and functions perfectly in the dark.

However, mmWave data is notoriously "noisy." The raw output, often visualized as a Range Doppler Map (RDM), varies significantly depending on the specific antenna used and the environmental noise floor. If you set a high threshold to filter out noise, you risk deleting the actual gesture signal. If the threshold is too low, the signal is buried in static. The authors tackle this "enigma" by finding a way to clean the data without destroying the underlying information.

Methods

The authors propose a "Neural Architectural Nonlinear Pre-Processing" approach. Instead of using traditional linear filters, they use a deep learning pipeline:

  1. Synthetic Noise Generation: To train a model to remove noise, you need to know what "clean" looks like. The authors created a synthetic dataset by taking "ground truth" images (gestures captured at a clear -90 dB threshold) and mathematically mixing them with pure noise samples (captured at -95 dB and -100 dB when no motion was present).
  2. U-Net (Denoising): This synthetic noisy data is fed into a U-Net. The U-Net acts as a nonlinear pre-processor, learning to map the messy, noisy RDM back to the clean ground truth.
  3. EfficientNet-B0 (Classification): The denoised output from the U-Net is then passed to EfficientNet-B0, a convolutional neural network (CNN) optimized for high accuracy with low computational overhead. This stage classifies the cleaned image into one of four gestures.
The end-to-end system model showing the flow from raw data to U-Net denoising and finally EfficientNet classification.

Datasets

The study utilized real-world data gathered using Infineon’s BGT60TR13C 60 GHz mmWave FMCW radar.

  • Gestures: The dataset consists of four hand movements: Left, Right, Away from radar, and Towards radar.
  • Noise Samples: Approximately 10,000 noise-only images were collected at different decibel thresholds to facilitate the synthetic noise generation process.
Fig. 1. Four way hand gesture recognition data set configuration method using mmWave radar. (From the left, Gesture 1: left, Gesture 2: right, Gesture 3: away from radar, Gesture 4: towards radar).

Key Insights and Ideas

  • Overcoming Antenna Discrepancy: One of the most interesting observations was that different antennas on the same device produce wildly different noise profiles for the same gesture. A standard hard-coded filter can't handle this variability, but a neural network can learn to be robust against these hardware quirks.
  • Synthetic Training as a Bridge: By intentionally "breaking" clean data with measured noise, the authors created a robust training set that allows the U-Net to generalize to real-world noise it hasn't seen before.
  • Nonlinear is Better: Traditional pre-processing is often linear. By using a U-Net, the system performs "nonlinear" pre-processing, which is much better at preserving the subtle edges of a hand gesture signal while aggressively suppressing background static.

Key Results

The results demonstrate that the denoising stage is highly effective. Visually, the U-Net is able to isolate the core gesture signal from a cloud of background noise.

Visual comparison of the Range Doppler Map before and after the U-Net noise rejection process.

Quantitatively, the model's perception accuracy converged to approximately 90% or higher over 50 training epochs. The authors note that this is a significant improvement over conventional methods that feed raw, noisy images directly into a classification CNN.

Limitations and Future Work

While the results are promising, the paper focuses on a relatively small set of four basic gestures. In a real-world cockpit, gestures might be more complex or subtle. Additionally, while the authors mention the suitability of this method for "on-driving" environments, the current study appears to be conducted in a controlled setting. Future research could test the model's performance against the intense vibrations and electromagnetic interference found in a moving vehicle on a busy road.

Summary updated

Abstract

In modern on-driving computing environments, many sensors are used for context-aware applications. This paper utilizes two deep learning models, U-Net and EfficientNet, which consist of a convolutional neural network (CNN), to detect hand gestures and remove noise in the Range Doppler Map image that was measured through a millimeter-wave (mmWave) radar. To improve the performance of classification, accurate pre-processing algorithms are essential. Therefore, a novel pre-processing approach to denoise images before entering the first deep learning model stage increases the accuracy of classification. Thus, this paper proposes a deep neural network based high-performance nonlinear pre-processing method.
BibTeX
@misc{baek2022neural,
  author = {Hankyul Baek and Yoo Jeong and Ha and Minjae Yoo and Soyi Jung and Joongheon Kim},
  title = {{Neural Architectural Nonlinear Pre-Processing for mmWave Radar-based Human Gesture Perception}},
  year = {2022},
  eprint = {2211.03502},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2211.03502},
}
Figure