A robot that learns by trial and error normally relies on someone to put the world back after each attempt. Reset-free reinforcement learning removes that person. A second "backward" policy learns to return the system to a sensible starting point, and a learned judge decides when that return has succeeded. Every such judge in the literature looks at the full physical state: the arm's joint angles, the peg's pose. Speech breaks that assumption. The tongue body, the tongue root, the soft palate and the pharynx are hidden inside the mouth. A learner, or a clinician, sees only the lips and hears only the sound, and the sound says nothing during silence. A reset judge that looks only at lips and audio can declare "back to neutral" while the tongue sits somewhere else entirely. The forward learner then starts its next attempt from a state it cannot see and was never trained on.
This note proposes to name and measure that failure, which we call aliased resets, and to fix it. The fix changes what a reset has to accomplish. Under occlusion, a good reset returns the hidden state to a familiar region and also makes the hidden state known again, for example by producing a short voiced probe sound whose acoustics pin down the tongue. We test this in a vocal-tract environment built from real articulography data, and we check that it generalizes on a standard reset-free manipulation benchmark with part of the state masked.
Core Claim
Existing reset mechanisms reward the backward policy for landing in states that resemble demonstration or initial states. MEDAL does this with a discriminator trained against expert demonstrations (A State-Distribution Matching Approach to Non-Episodic Reinforcement Learning). Example-based resets use a recursive classifier over initial-state examples (Automating Reinforcement Learning With Example-Based Resets). RISC hands control between policies using a success critic (Intelligent Switching for Reset-Free RL). When these judges see only observations that map many hidden states to the same reading, they accept resets whose hidden part is off-distribution. Our claim is that this aliasing, not exploration difficulty, is the dominant cause of reset-free learning collapse in occluded systems.
We further claim the failure is fixed by moving the reset objective from observations to a belief, meaning an estimate of the hidden state together with its uncertainty, and by penalizing uncertainty directly. Concretely, the backward policy is rewarded by
$$r_b(b_t) = \log D(\mu_t) - \beta\, H(b_t),$$
where $b_t$ is the belief over hidden articulator positions at time $t$, $\mu_t$ is its mean, $D$ is a discriminator scoring how demonstration-like a hidden state is, $H(b_t)$ is the belief's entropy, and $\beta$ trades off the two terms. Handoff to the forward policy is allowed only when $H(b_t)$ falls below a threshold. We predict that the entropy term alone causes the backward policy to discover active-sensing gestures: brief voiced sounds that make the tongue observable before handing back control.
The belief estimator is trained offline on paired data where the hidden state was measured. At deployment it never sees the hidden state, which matches the "missing modality at inference" setting that Incorporating Ultrasound Tongue Images for Audio-Visual Speech Enhancement addresses with a lip-to-tongue memory. That paper uses the cross-modal estimate passively to clean audio. We use a comparable estimate to decide when control can safely resume.
Why this intersection, and what is not new
The research file reports no paper combining occluded articulators with reset-free learning. The adjacent work shows why each half leaves the problem open.
Two papers supply the environment. Articulation GAN drives a frozen articulatory synthesizer, ema2wav, trained on mngu0, which turns 12 articulator coordinates plus a voicing channel into audio. It learns trajectories only through acoustic realism, in fixed episodic windows with no sequential decisions. Speech robot mimicking human articulatory motion replays human articulography open-loop on a physical talking robot. It reports resorting to heuristic estimates for the posterior tongue points, and a jammed configuration still needs manual realignment. Together they show the vocal tract is a controllable plant whose hard part is the hidden back of the tongue.
Two further papers motivate the claim that the aliasing is real. Combining audio and video recovers tongue coils at only about 0.7 mean correlation, and the back of the tongue is worst (Reconstructing tongue movements from audio and video). Covert, silent lingual gestures occur and leave no acoustic trace (Characterizing Covert Articulation in Apraxic Speech Using real-time MRI). So during silence the hidden state can move while observations stay constant.
The reset-free work supplies the method and the baselines. The benchmark formalism in Autonomous Reinforcement Learning: Formalism and Benchmarking assumes reachable, observed states throughout its benchmark tasks. MEDAL and RISC reuse that assumption. We keep their forward–backward structure and change what the backward policy optimizes and when handoff happens.
Research Program
We would build three components.
The first is an occluded vocal-tract environment. The state is the mngu0 articulator configuration: lips, jaw, and tongue tip, body and dorsum (Announcing the electromagnetic articulography (day 1) subset of the mngu0 articulatory corpus). Actions are velocity commands passed through second-order dynamics with process noise, so hidden coordinates drift when unobserved. We will be explicit that these dynamics are our construction. Observations are the ema2wav audio, which carries the equifinality ambiguity the synthesizer inherits from real data, plus the lip and jaw coordinates. The tongue coordinates are withheld from the agent but logged for evaluation. The forward task is producing target words from the eight-word TIMIT subset used by Articulation GAN. The reward comes from a frozen phone recognizer on the synthesized audio. Demonstrations are 10 to 30 mngu0 utterances, matching MEDAL's demonstration budget. Training runs as one unbroken stream, and a simulated "human intervention" is counted whenever the hidden state leaves the physiological range seen in mngu0.
The second component is the belief estimator. It is a recurrent inversion network trained offline on mngu0 audio and lip data to output a Gaussian over tongue coordinates. It is calibrated on held-out utterances so that its entropy tracks actual error.
The third component is the belief-based reset agent. It uses the reward above, uncertainty-gated handoff, and RISC's non-terminal bootstrapping across switch points, which avoids the bias RISC documents from treating handoffs as terminal.
Baselines
The naive reference point is forward-only learning with no backward policy. It shows how far unaided drift degrades learning.
The strongest existing method is MEDAL, run with an observation-based discriminator. We also run RISC's success-critic switching on the same observations. On EARL's Sawyer-Peg task, MEDAL is reported at 40% success against 0% for earlier autonomous methods (A State-Distribution Matching Approach to Non-Episodic Reinforcement Learning). For RISC, the file reports only that it matched or exceeded the best known results on 3 of 4 EARL tasks (Intelligent Switching for Reset-Free RL). No reported result was found on Lacuna for any reset-free method on an articulatory environment, so on the vocal tract every number will come from our own runs. Fairness rules are fixed in advance. Every method gets the same demonstrations, the same environment steps, the same forward learner and network sizes, and the same hyperparameter search budget, and every method is run over the same seeds.
We also include a history-based variant of MEDAL: an LSTM discriminator over past observations and actions. It is the direct test of whether memory alone removes aliasing.
Two privileged references bound the problem. The first is MEDAL with a discriminator that sees true tongue state. It is an upper bound that is unattainable in deployment. The second is episodic training with teleport resets, the standard oracle in EARL.
Two ablations isolate the key ingredient. One keeps belief-mean matching but drops the entropy term and the uncertainty gate. The other keeps the gate but matches observations. Beating MEDAL means higher forward success and fewer simulated interventions at equal steps, with the gap closing most of the distance to the privileged-discriminator bound.
Why not just…?
Why not just give the reset discriminator memory? This is the most serious threat. If the action history is informative and drift is small, an LSTM can dead-reckon the tongue position and aliasing disappears. We will not argue past this; we will measure it. We sweep the process-noise magnitude and the length of silent intervals. Our method should matter only where noise accumulates during silence faster than history can track it. If the history-based variant matches us across the whole sweep, the conclusion is that memory suffices, and the contribution shrinks to the diagnostic metric.
Why not just command a hard-coded neutral posture? An open-loop "go to rest" is the analogue of the kinematic replay in Speech robot mimicking human articulatory motion. It works when the dynamics are known and noiseless, and we include it as a baseline. With process noise it cannot correct hidden drift, because it never observes that drift. The sweep will show the noise level at which it breaks.
Why not just train with teleport resets in simulation? EARL reports that oracle-reset policies lost 37.4% of their performance from randomized starts on Tabletop-Organization, against 2% for the autonomous VaPRL (Autonomous Reinforcement Learning: Formalism and Benchmarking). We will test whether that brittleness holds for articulation by starting each trained policy from drifted hidden states. We will also note that physical talking hardware cannot teleport, so this baseline does not transfer to hardware.
Why not just use a better passive inversion model? Offline inversion reaches about 0.99 mm error on mngu0 (Announcing the electromagnetic articulography (day 1) subset of the mngu0 articulatory corpus), but that figure is measured on voiced speech. Our belief estimator is such a model. The question is whether its estimates are reliable at the handoff moments that matter, which are often silent. We report estimator error separately for silent and voiced frames.
Decisive Experiments
The central measurement is the aliased-reset rate: the fraction of accepted resets whose logged hidden state lies outside the support of the demonstrations. On the vocal tract, the first check is whether observation-based MEDAL and RISC show a high aliased-reset rate, and whether that rate predicts their forward-success deficit across seeds and noise levels. If aliasing does not correlate with failure, the core claim is wrong.
The second check is whether our agent lowers the aliased-reset rate and intervention count toward the privileged bound. The entropy ablation should show that the uncertainty penalty is responsible for the improvement.
The third check is behavioral. We ask whether the learned backward policy emits short voiced probes before handoff, and whether these probes disappear when the tongue is made observable. That control separates active sensing from incidental vocalization.
For generality, we repeat the comparison on EARL's Sawyer-Peg and Tabletop-Organization with the object pose masked from the reset judge. There, MEDAL's reported 40% Sawyer-Peg success under full state is the reference: masking should push it down, and our method should recover most of the lost success. A secondary outcome is word intelligibility, compared against the 72% Articulation GAN reports (Articulation GAN). It asks whether closed-loop, reset-free practice narrows the intelligibility cost of the articulatory bottleneck. This comparison is only indicative because the evaluation protocols differ.
References
- A State-Distribution Matching Approach to Non-Episodic Reinforcement Learning
- Automating Reinforcement Learning with Example-based Resets
- Intelligent Switching for Reset-Free RL
- Incorporating Ultrasound Tongue Images for Audio-Visual Speech Enhancement
- Articulation GAN: Unsupervised modeling of articulatory learning
- Speech robot mimicking human articulatory motion
- Reconstructing tongue movements from audio and video
- Characterizing Covert Articulation in Apraxic Speech Using real-time MRI
- Autonomous Reinforcement Learning: Formalism and Benchmarking
- Announcing the electromagnetic articulography (day 1) subset of the mngu0 articulatory corpus