Cancelling a Robot's Own Noise So Contact Sound Transfers From Human Demonstrations

When a robot's fingers close around a part to insert it, the cameras stop seeing what matters. The gripper hides the object, and the socket hides the part's tip. Contact microphones are cheap piezo discs glued to the fingers that record vibrations at audio rates. They still carry the evidence of a tip catching a rim, sliding, or clicking home. But on a real robot those microphones mostly record the robot itself: motor whine, gear chatter and link resonance. This self-generated noise changes with every motion, and it reaches the sensor through a nonlinear mechanical path. Human demonstrations recorded with the same sensor contain almost none of it, because a person's hand is soft and quiet. So the contact-sound channel a policy learns from human data looks different from the one it hears on the robot. The mismatch is worst exactly during the occluded final phase, when sound is supposed to take over from vision.

Speech engineers solved a close cousin of this problem. A phone's microphone hears the phone's own loudspeaker, distorted by a cheap amplifier. Acoustic echo cancellation removes it because the device knows exactly what it played. The proposal is to treat the robot the same way. The robot knows its own commanded torques and joint velocities, and those commands are the "far-end reference" of its self-noise. A hybrid canceller, with a linear stage plus a neural stage for the nonlinear remainder, can subtract the robot's echo of its own actuation from the contact microphone. What remains should be a contact-sound signal that looks like the human-collected one. A policy trained on human demonstrations could then use it as closed-loop feedback while the camera is blind.

Core Claim

Robot ego-noise on fingertip contact microphones is a nonlinear echo of a reference the robot already knows. Cancelling it with a reference-conditioned residual suppressor does two things at once. It turns contact audio into a modality that transfers from human handheld demonstrations to the robot without per-task real-robot fine-tuning. It also makes that audio usable as feedback during the visually occluded terminal phase of insertion-style tasks. The mechanism comes from echo cancellation, the embodiment gap is the transfer problem it addresses, and visual occlusion is the regime where the payoff is measured.

The signal model makes this concrete:

$$m(t) = (g * c)(t) + \big(h * f(u(t))\big)(t) + n(t)$$

Here $m(t)$ is the fingertip microphone signal and $c(t)$ is the contact excitation we want. The term $g$ is the short path from contact point to sensor, and $u(t)$ is the robot's known actuation reference, meaning commanded joint torques and joint velocities. The function $f$ is the unknown nonlinear transformation that motors, gears and backlash apply to that reference. The term $h$ is the linear structural path from actuators to sensor, and $n(t)$ is external noise. This is the same structure as the nonlinear echo model in telecommunications. The far-end playback signal is replaced by $u(t)$, and the near-end talker is replaced by contact. Purely linear adaptive filters cannot remove the $f$ component. That is the central finding behind DNN-based residual echo suppression, and it motivates a neural second stage.

Why This Is Not Already Done

Two lines of work put contact microphones on manipulators. Hearing Touch mounts four piezo microphones on a Franka gripper and fuses audio encoders pretrained on AudioSet with R3M vision features. It beats the next baseline by 23% in success across flipping, scooping and zipping. Its authors note the system suffers in noisy conditions, and it applies no cancellation of any kind. Its robustness tests vary visual distractors rather than physical self-occlusion. MimicTouch records human demonstrations with a wearable rig that includes a contact microphone, compresses the audio with self-supervised learning, and transfers it to a robot. The offline policy reaches 40% success and needs about 3 hours of on-robot residual reinforcement learning to reach 96%. Neither paper models the robot's self-noise.

Both papers motivate this proposal. We do not claim that the acoustic mismatch is the reason MimicTouch needs fine-tuning, since its gap also includes pose and tactile differences. Testing that is part of what we would measure.

The tactile occlusion papers constrain the design rather than supply it. Touch in the Wild shows that a portable visuo-tactile gripper lifts occluded test-tube collection from 25% to 85% success. DECO adds a tactile adapter that drives the blind phase of humanoid assembly, a 20-point absolute gain. Both use pressure arrays sampled at tens of hertz, not structural sound, and neither has to deal with self-noise.

On the method side, the echo-cancellation literature supplies the architecture. Deep Residual Echo Suppression and Noise Reduction: A Multi-Input FCRN Approach and TasNet-MI show that a neural suppressor works far better when it is fed both the linear filter's residual and its echo estimate, not the raw mixture. TasNet-MI exceeds 45 dB of echo return loss enhancement (ERLE, how many decibels of echo are removed) on speech, against about 29 dB for a single-stream network, at 15 ms latency. Deep Multitask Acoustic Echo Cancellation shows that predicting the echo as an auxiliary task helps under nonlinear distortion and noise.

All of these assume a sample-rate digital reference. Our reference arrives at roughly 1 kHz from joint controllers, while the microphone samples at 32–48 kHz, and it passes through mechanics rather than an amplifier. Bridging that rate and physics gap is new. So is using cancellation as an embodiment-alignment step for policy learning.

Research Program

The canceller. The first stage is an order-tracking linear canceller. From each joint's velocity and known gear ratios we synthesize sinusoidal bases at the motor and gear-mesh harmonic frequencies. A normalized least-mean-squares filter, the standard adaptive filter used in echo cancellation, fits their amplitudes and phases online. This removes the stationary, speed-locked part of the self-noise.

The second stage is a time-domain residual suppressor modelled on TasNet-MI. It takes three streams: the linear residual, the linear stage's noise estimate, and an embedding of the upsampled reference $u(t)$, which carries commanded torques, velocities and accelerations. It outputs an estimate of the contact component. The streams are kept separate up to subtraction-style fusion blocks, as in TasNet-MI. Following Deep Multitask AEC, the network also predicts the ego-noise as an auxiliary target.

The latency budget is set by the control loop. We target under 20 ms, comparable to TasNet-MI's 15 ms.

Training without labels. The robot provides its own supervision. We record two kinds of data. Ego-noise-only data comes from the robot executing many randomized and retargeted task-like trajectories in free space, so there is no contact. Contact-only data comes from a static rig: the instrumented finger is held by the robot with its brakes engaged while objects are pressed, slid, tapped and inserted against it by a second fixture or by hand.

Summing these gives mixtures with exact ground truth, just as echo-cancellation benchmarks mix echo and near-end speech. Piezo sensors also saturate on hard impacts, which a digital sum would miss. To reproduce this, we apply each sensor's measured clipping curve after mixing and add a flag marking clipped samples. This is a simplification, and the real-hardware tests below check it.

Transfer and use under occlusion. Human demonstrations are collected with a handheld copy of the robot's fingers in the style of Touch in the Wild. The handheld fingers carry the identical piezo sensor and a wrist camera. The finger geometry and sensor are shared, so what remains between human data and robot data is mostly the acoustic embodiment gap: self-noise, link resonance and saturation behaviour. The kinematic gap studied by DexMV and Im2Flow2Act is largely set aside on purpose. Uninstrumented human video is out of scope.

The policy is the Hearing Touch architecture, with AVID audio and R3M vision features and transformer cross-attention, trained only on the human demonstrations. At deployment it receives cleaned robot audio.

Baselines

The naive reference point is the same policy deployed on raw robot microphone audio, with no cancellation. A second reference point is vision only. Together they show how large the acoustic gap is and how much audio matters once vision goes blind.

The strongest existing method is the Hearing Touch policy itself, with AudioSet-pretrained audio, R3M vision and cross-attention fusion. It is the published system closest to our setting and is the architecture we build on. We run it in two ways. In the first it is trained on human handheld data and deployed raw. In the second it is trained on 40–60 robot-collected demonstrations per task, the regime its paper reports. The second variant is the "just collect robot data" ceiling.

The file gives no per-task success numbers for Hearing Touch. What it reports is a 23% average success gain over the next best baseline, a 76% reward gain, and a drop under visual distractors of only 20% against vision-only's 60%. We will reproduce it on our tasks rather than quote it.

For cancellation quality, the strongest acoustic baseline is TasNet-MI fed only the microphone residual and the linear estimate, without the reference embedding. Its reported ERLE exceeds 45 dB on speech, but no reported result was found on Lacuna for ego-noise cancellation on robot contact microphones.

The ablation that isolates the key ingredient gives the policy the same reference streams $u(t)$ as extra inputs but performs no cancellation. Any gain must therefore come from the explicit removal of self-noise, not from the information in the reference.

Fairness rules apply to every policy comparison. All policies share the backbone, the human demonstration set and the hyperparameter search budget, and the canceller's compute is counted inside the latency budget.

Beating the baselines means two things. First, the human-trained policy on cleaned audio must match or exceed the robot-trained Hearing Touch variant on success in the occluded terminal phase. Second, it must clearly beat both the raw-audio deployment and the reference-as-input ablation, judged with confidence intervals over at least 20 trials per task.

Why not just…?

Why not just notch or bandpass filter the motor frequencies? Fixed filtering removes stationary hum. But gear-mesh and motor harmonics move with joint speed and sweep across the same band as scraping and click transients. Deep Multitask AEC also documents linear processing collapsing under nonlinear distortion plus noise, from 51.61 dB to 10.13 dB ERLE. Our first stage is in fact a speed-tracking linear canceller. Running it alone, without the neural stage, is the direct test. If it closes most of the gap, the honest conclusion is that the neural stage is unnecessary, and we would report that.

Why not just feed proprioception to the policy and let it learn to ignore the noise? This is a real threat, and it is the reference-as-input ablation above. The argument against it rests on data. The human demonstrations contain no self-noise, so the policy never sees the noise-to-reference relationship during training and has nothing to learn invariance from. Separately, DECO and Hearing Touch both find that loosely structured fusion lets noisy auxiliary signals degrade performance. If the ablation matches our method when trained on robot data but fails when trained on human data, the transfer claim stands.

Why not just add recorded robot noise to the human audio during training? This is the strongest alternative. We can replay retargeted human trajectories on the robot in free space to get motion-matched self-noise and mix it into the human recordings. It needs the same recordings our canceller does. We will run it as a full baseline and also combined with cancellation.

Augmentation teaches the policy to be invariant to noise. Cancellation removes the noise at test time, and it also resolves contact events masked by noise that is louder than the contact. Our prediction is that cancellation wins on faint contacts such as micro-slip and latch clicks, while augmentation is competitive on loud impacts. If augmentation matches cancellation everywhere, the contribution shrinks to the acoustic analysis and we will say so.

Why not just use tactile skins or more cameras? Touch in the Wild and DECO already show that pressure arrays help under occlusion, at 23 Hz and below. Extra cameras cannot see inside a socket. We treat a matched tactile sensor as a complementary comparison on one task rather than a competitor, because the question here is specifically whether sound survives the embodiment gap.

Proposed Experiments

Offline acoustic evaluation. On held-out synthetic mixtures and on real recordings, we report ego-noise reduction in dB, the robot analogue of ERLE, and contact-onset timing error against labels derived from a force sensor. We also report contact-event detection F1 at noise-to-contact ratios from strongly negative to positive. These metrics are reported separately for clipped and unclipped events, to test whether the clipping simulation holds on real hardware.

Distribution alignment. We train a small classifier to tell human-demonstration audio embeddings from robot audio embeddings. Its accuracy should fall from near 100% on raw robot audio towards chance on cleaned audio. If it does not, the transfer premise fails before any policy is run.

Policy evaluation. Tasks are chosen so that the terminal phase is occluded. They include test-tube insertion into a rack, following Touch in the Wild; plug-into-socket; snap-fit latching; and pouch zipping from Hearing Touch. A controlled condition additionally blanks camera frames once the gripper is within a few centimetres of the target, which isolates the occluded window.

Success is measured overall and within the occluded window. We also measure the number of real-robot demonstrations needed for the robot-trained baseline to match the human-trained cleaned-audio policy. For context, MimicTouch's offline 40% and its 3-hour fine-tuning to 96% are the published human-to-robot reference points. Their tasks differ, so they are not a head-to-head comparison.

Decisive checks. The proposal succeeds if three things hold. Cleaned audio drives the human-to-robot classifier towards chance. The human-trained policy on cleaned audio matches the robot-trained Hearing Touch variant in the occluded window. And it beats both the reference-as-input ablation and the motion-matched augmentation baseline on faint-contact tasks.

It fails cleanly if augmentation or proprioceptive conditioning alone does as well. In that case cancellation is not needed for transfer, and contact audio's embodiment gap is a training-distribution problem rather than a signal problem.