Virtual Flesh Points: Near-Field Radio Focusing for Contactless, Open-Vocabulary Silent Speech

Silent speech interfaces turn mouthed but unvoiced speech into text. The ones that work best today rely on wired coils glued to the tongue and lips, and even these recognize only about 25 words. Contactless radar avoids the wires, but it sees the whole face as one blurred reflector, so head motion swamps the signal and vocabularies stay near a dozen words. We propose a palm-sized millimeter-wave antenna array held 15–40 cm from the face. At that distance the face sits deep inside the array's near field, the zone where incoming wavefronts are curved rather than flat. In that zone the array can concentrate its sensitivity on a small 3D spot, not just a direction. Aiming it at a handful of spots on the lips, jaw, cheeks and under-chin skin gives a set of "virtual flesh points": streams of micro-displacement that play the role of glued-on coils without touching the speaker.

These streams are then decoded with two deliberate choices. The first is a small, fixed recurrent network whose only trained part is a linear output layer, which can be recalibrated in closed form within seconds for a new user or session. The second is an output made of sub-word phonological units rather than whole words, so that words never seen in training can be recognized through a lexicon and language model. The combined question is whether contactless sensing can reach the accuracy of the wired systems at under 300 ms latency, and then pass their vocabulary ceiling.

Core Claim

Near-field focusing supplies the one thing contactless speech radar lacks: resolution in depth at close range that does not depend on bandwidth. The proposal claims three things. First, a focused array yields per-location displacement signals whose information content approaches that of electromagnetic articulography (EMA, the wired tongue/lip coil tracking method). Second, these signals are low-dimensional and slow enough that a fixed-dynamics decoder with a linear readout matches trained recurrent decoders while being far cheaper to recalibrate. Third, decoding into phonological features lets vocabulary grow without per-word training data.

The physics supports the first claim at realistic scale. The near-field boundary is the Rayleigh distance

$$R = \frac{2D^2}{\lambda},$$

where $D$ is the array aperture and $\lambda$ the wavelength. For a 10 cm aperture at 77 GHz ($\lambda \approx 3.9$ mm), $R \approx 5$ m. A face at 30 cm is therefore an order of magnitude inside the regime where, according to Near-Field Communications: A Comprehensive Survey, the quadratic phase term across the aperture lets energy be focused at a specific range and angle. Points at the same angle but different depths can then be separated.

The motivating failure comes from RaSSpeR: Radar-Based Silent Speech Recognition, which places a single mmWave radar board about 15 cm from the face. Because range readings were corrupted by head motion, it had to discard them and use only velocity-dispersion features. Its authors explicitly name multi-antenna beamforming as the missing piece.

The closest sensing work is A Spatio-Temporal-Frequency Transformer Framework for Near-Field Target Recognition. It shows that near-field channel tensors carry fine geometric signatures, but only for binary car-versus-motorcycle recognition on synthetic, offline, rigid targets. We instead track deforming biological surfaces in a streaming setting, with a vocabulary-scale output.

Research Program

The system has three stages, each of which can be tested on its own.

Stage 1: sensing. An FMCW (frequency-modulated continuous-wave) MIMO array, built from cascaded commercial mmWave transceivers to form a roughly 10 cm virtual aperture, computes near-field focusing weights for a grid of candidate 3D points over the lower face. It tracks the unwrapped phase at each focal point, which converts to sub-millimeter displacement. A lightweight head tracker, re-estimated from the strongest static reflectors every few hundred milliseconds, rigidly moves the focal grid with the head. This is the direct answer to the head-motion failure in RaSSpeR. A small subset of focal points (about 8–16) is selected per user by maximizing speech-band energy during a one-minute calibration. That subset is the "virtual flesh point" set.

Stage 2: decoding. The focal-point streams drive an echo state network, meaning a fixed random recurrent network kept stable by scaling its weights. The only trained part is a ridge-regression readout,

$$W_{\text{out}} = Y X^{\top}\,(X X^{\top} + \beta I)^{-1},$$

where $X$ stacks reservoir states over time, $Y$ holds frame-level phonological targets (place, manner, voicing, lip rounding), and $\beta$ is a regularization weight.

We borrow the "domain-informed poles" idea from When Learning Hurts: Fixed-Pole RNN for Real-Time Online Training. That paper shows that fixing recurrent poles avoids the ill-conditioned optimization of trained poles, with a Hessian condition number around 2.5 versus roughly $10^3$, in online wireless symbol detection. Here the poles are placed to cover articulatory time constants (movements below about 15 Hz) rather than drawn at random. The same paper constrains the claim: its proofs cover only linear single-input systems and online training. We therefore treat the reservoir's advantage over an offline-trained recurrent net as an empirical question, not an inherited result.

Following the transient-readout principle of A Model for Real-Time Computation in Generic Neural Microcircuits, the readout emits a phonological-feature posterior at every frame. The decoder never waits for a word to finish.

Stage 3: vocabulary. Frame posteriors are mapped to phoneme posteriors and passed to a streaming lexicon-constrained beam search with a small language model. This is the sub-word bottleneck strategy used by Large-Scale Visual Speech Recognition and SPEAK YOUR MIND! Towards Imagined Speech Recognition with Hierarchical Deep Learning, now applied to contactless RF sensing for the first time.

We expect the method to hit a physical limit. Millimeter waves barely penetrate tissue, so the tongue is visible only indirectly, through jaw, cheek and under-chin surface motion. Confusions between sounds that look alike at the surface, such as /p/, /b/ and /m/, will persist and must be resolved by the language model. Measuring exactly which phonological features the array can and cannot recover is itself a planned output.

Data. Before hardware collection, we will build a simulation. Real articulatory trajectories from the mngu0 EMA corpus will drive moving surface scatterers on a face mesh, producing synthetic near-field channels. This lets focusing, grid selection and decoder design be developed against a known ground truth.

We will then collect a new corpus. It will contain 10 speakers mouthing the 25-word command set used in the EMA studies, a 500-word held-out lexicon, and read phrases. A subset will be recorded with simultaneous EMA, which allows a direct test of how well each virtual flesh point stands in for a physical one.

Baselines

Naive reference. The do-nothing reference point is a faithful reimplementation of RaSSpeR on our recordings: a single virtual channel, velocity-dispersion features, and a bagging classifier. RaSSpeR reports average accuracy 0.826 on its 13-word European Portuguese corpus across 3 speakers. We will run it on our 25-word set with identical splits. It must be clearly beaten, by more than 10 points absolute, for the array to be worth its cost.

Strongest sensing competitor. The strongest competitor for the sensing claim is conventional MIMO FMCW processing: range bins from bandwidth combined with far-field angle steering, on the same hardware and data. This competitor is dangerous because FMCW already provides some range information. Our ablation keeps identical focal-point locations and the identical decoder, and swaps near-field focusing weights for far-field steering plus range gating. That isolates the quadratic-phase ingredient. Beating it means lower phoneme error at matched point count, especially under deliberate head motion.

Strongest accuracy and latency reference. For words, the strongest reference is the wired EMA line of work. Word Recognition from Continuous Articulatory Movement Time-series Data using Symbolic Representations, which applies symbolic discretization and an SVM over sliding windows, reports 80.00% online speaker-dependent accuracy on 25 words at 302 ms average latency. Whole-Word Recognition from Articulatory Movements for Silent Speech Interfaces, an RBF-kernel SVM on raw EMA, reports 93.71% isolated-word accuracy and 0.79 s prediction latency. We will also run both methods on our own synchronized EMA channels, so wired and contactless sensing are compared on the same utterances. Matching 80% online accuracy at 300 ms or less without contact would be the headline result.

Strongest learned-decoder competitor. On the decoder side, the competitors are a causal GRU and a small causal Conformer. Both are trained offline with full backpropagation and fine-tuned per user. We also include a causal adaptation of the STF Transformer from A Spatio-Temporal-Frequency Transformer Framework for Near-Field Target Recognition, run directly on focal-point channel tensors. All of these receive the same inputs, the same calibration data per new user (1, 5 and 15 minutes), and a matched hyperparameter search budget. Latency is measured on the same embedded processor.

Decoder ablations. One ablation swaps domain-informed poles for random poles. Another swaps phonological outputs for whole-word outputs, which isolates the bottleneck.

Why not just…?

Why not just use FMCW range bins with ordinary angle steering? This is the most serious alternative. Range resolution from a 4 GHz sweep is about 3.75 cm, which is coarse relative to lip-jaw-chin spacing. Angle alone cannot separate points along the same ray, as the near-field survey explains. Still, combining range bins with angle might be enough. The matched ablation settles it. If far-field processing with range gating equals focusing, we will report that, and the contribution reduces to the decoding and vocabulary stages.

Why not just train a causal GRU offline and fine-tune it per user? Fixed-pole evidence covers online training (When Learning Hurts), not offline training, so a GRU may well match reservoir accuracy. The reservoir's claimed advantage is narrower: recalibration in seconds via one linear solve, deterministic per-frame cost, and no gradient computation on the device. The calibration-budget sweep will show whether this matters. If the GRU wins at every budget with comparable adaptation time, we will adopt it.

Why not just classify whole words directly, as RaSSpeR and the EMA systems do? Whole-word classifiers cannot score words they never saw. The EMA work reports that continuous matching without discretization generated 8.08 false positives per sequence, versus 3.04 with symbolic discretization (Word Recognition from Continuous Articulatory Movement Time-series Data using Symbolic Representations). The held-out 500-word test is where whole-word models have no answer. On the closed 25-word set, however, whole-word models may win, and we will report both.

Why not just use a large pretrained sequence model? Models such as MEG-XL, which relies on 150-second contexts, and the 22-million-parameter Large Brain Language Model buy accuracy with context and compute that conflict with a sub-300 ms budget. No pretraining corpus exists for near-field face RF in any case. The causal Conformer baseline is our test of how much capacity helps at fixed latency.

Proposed Experiments

The first decisive check happens in simulation, driven by mngu0 trajectories. Can displacement recovered at focused points predict the underlying EMA coil trajectories? We will report RMSE in millimeters against the ground-truth coils. For context, the mngu0 announcement reports 0.99 mm acoustic-to-articulatory inversion error. This is an inversion from sound, not a like-for-like target, but it indicates what "close to EMA" means.

The second check uses the real corpus. We measure phoneme and phonological-feature error under three conditions: still head, natural head motion, and scripted head motion. Each condition compares near-field focusing, far-field steering with range gating, and single-channel RaSSpeR features. The success criterion is that focusing's advantage grows with head motion.

The third check measures streaming word accuracy on the 25-word set against the 80.00% and 302 ms EMA reference. Latency is decomposed into algorithmic lookahead, compute time per frame, and the offset from articulation end to word emission.

The fourth check measures zero-shot word error on the held-out lexicon as vocabulary grows from 25 to 100 to 500 words. It also produces a confusion analysis naming which phonological features the RF surface view cannot recover.

The fifth check measures cross-session and cross-user recalibration: accuracy versus calibration minutes and versus wall-clock adaptation time, comparing the reservoir readout against the fine-tuned GRU and Conformer.

The proposal succeeds if two things hold. Near-field focusing must beat the matched far-field ablation under head motion. And the contactless system must reach the wired EMA 25-word online accuracy at comparable latency while decoding unseen words above chance-with-lexicon.