Hack-Resistant Acoustic Rewards for Contact-Rich Robot Learning

Many robot assembly tasks announce their own success out loud: a USB plug seats with a click, a card snaps into a slot, a latch catches. A cheap contact microphone taped to the gripper can hear these events even when the robot's own hand blocks the camera. That makes sound an attractive replacement for hand-written reward equations in reinforcement learning (RL), the setting where a robot improves by trial and error guided by a numeric score. The catch is that a learned reward is itself a target for optimization. A policy rewarded for "sounds like a click" will, given enough trials, find cheaper ways to make click-like sounds. It might tap the connector against the housing, scrape the socket edge, or drive its motors at the speed whose gearbox whine happens to fool the detector. This is the familiar reward-hacking failure of hand-crafted rewards (Active Reward Learning), moved into the audio channel.

This note proposes to treat the acoustic reward detector the way sound-event-detection researchers treat a rare-event classifier. We build its training set by synthetic mixing: isolated target sounds are overlaid on varied backgrounds at random signal-to-noise ratios, so the model cannot rely on shortcuts. The new step is where the backgrounds and hard negatives come from. Beyond ambient noise banks, we continuously harvest audio from the current policy's own failed episodes and from the robot's motor noise, and mix real success clicks into it. The detector is retrained against exactly the sounds the policy is learning to produce. The claim is that this policy-in-the-loop augmentation keeps an acoustic reward trustworthy under optimization pressure, where standard ambient-noise augmentation does not. Combined with a visual reward for the approach phase, it should also make contact-rich RL robust to the lighting and occlusion failures that break vision-only rewards.

Core Claim

The claim has three parts. First, an acoustic success detector trained with conventional augmentation will be exploited during RL: the fraction of reward-firing events that correspond to real success will fall as training proceeds. Second, adding hard negatives harvested from the policy's own failed episodes, together with recorded motor noise, into the synthetic mixing pipeline prevents that decline. Third, the resulting reward matches a strong visual reward in nominal lab conditions and clearly beats it under glare, occlusion, and background changes.

The supervision cost is one bit per episode, success or failure, drawn from an electrical or human check at episode end. This is far cheaper than the language subtask annotations that SARM needs across hundreds of demonstration hours (SARM).

Relation to Prior Work

Four lines of work sit closest to this proposal.

The Intrinsic Sound Curiosity Module (Impact Makes a Sound and Sound Makes an Impact) uses simulated impact sounds in ThreeDWorld as a curiosity bonus: the agent is rewarded when a contact sound surprises it. The audio is used only during pretraining and is then switched off. It is also pristine simulated audio. We instead use sound as a task reward that stays on during policy learning, and we assume noisy real audio.

Hearing Touch (Hearing Touch) shows that piezo contact microphones plus an audio encoder pretrained on AudioSet make imitation policies far more robust to visual distractors. It never builds a reward. Its authors also note that loud ambient noise and motor noise are untested. We reuse their sensor setup and pretrained encoder, and their stated limitation is the exact regime we target.

Focus-Then-Contact (FTC) replaces hand-written rewards with a visual distance to a goal keyframe for a small residual RL policy. Its authors concede two weaknesses. The reward degrades under glare and occlusion. It also punishes non-monotonic maneuvers, such as pulling back to realign a jammed plug, because those moves increase the distance to the goal image. FTC is our strongest baseline and also the approach-phase component we build on.

MSDP (Self-Supervised Multisensory Pretraining) fuses force-torque sensing into RL representations but keeps hand-engineered rewards, and it names acoustic sensing as future work.

The augmentation method itself comes from the sound-event-detection literature. Overlaying isolated rare events onto diverse background scenes at varied SNRs prevents overfitting (Rare Sound Event Detection Using 1D CRNNs), as does stochastic polyphonic mixing (Bidirectional LSTM-HMM Hybrid System). This mixing recipe was standardized as a benchmark pipeline in DCASE2017. What those works lack is an adversary: their background distribution is fixed. In RL, the "background" is partly generated by a policy that is actively searching for detector errors. Our proposal changes exactly this one thing.

Two works constrain the design. The ObjectFolder benchmark (The Object Folder Benchmark) reports a persistent sim-to-real gap for synthesized contact audio. That warns against relying on modal synthesis (ObjectFolder 2.0, ThreeDWorld) for the positive examples. Accordingly, our main pipeline synthesizes mixtures from real clip exemplars, and fully simulated clicks appear only as a secondary question.

Research Program

Sensing and detector. The robot carries piezo contact microphones as in Hearing Touch. The reward detector is a small classifier head on a frozen audio encoder pretrained on AudioSet, the AVID encoder used by Hearing Touch. It outputs, for each short audio window, the probability that the task's completion event just occurred.

Seed data. Positive exemplars are windows around the completion click, cut from the 20 to 30 teleoperated demonstrations already needed to train the base imitation policy in the FTC recipe. Each positive is timestamped from the demonstration's end.

Three background banks. Every training example is built by overlaying a positive or negative exemplar onto one or more background tracks, at a randomly drawn signal-to-noise ratio and with a random gain and filter that mimic different microphone mountings. The backgrounds come from three banks:

  1. An ambient bank of room and lab recordings, including speech and nearby machinery.
  2. An ego-noise bank recorded while the arm executes random motions at varied speeds, with and without free-air contact.
  3. A harvested bank, which grows during RL. After each failed episode, confirmed failed by the episode-end check, every window where the current detector fired or nearly fired is added as a hard negative. The rest of that episode's audio joins the background pool.

Online retraining. The detector is retrained every few minutes of robot time on a mix drawn from all three banks. To keep it from forgetting that real clicks happen amid real policy noise, positives are also mixed onto harvested failure audio, not only onto ambient noise.

Reward composition. The reward adds FTC's visual keyframe progress term to an acoustic term. The acoustic term pays out when the detector fires. We also give a smaller reward for detected first contact, which gates the visual term off once the robot is touching the part. This lets the policy pull back and realign without being punished by a rising visual distance.

Secondary question. We ask whether modal-synthesized clicks can substitute for the real positive exemplars. To do so, we randomize modal frequencies and damping in a ThreeDWorld-style renderer and train the detector on these synthesized positives.

Measuring reward hacking. The central measurement is reward precision under optimization: over a sliding window of training, the fraction of acoustic reward events that coincide with oracle-verified success.

Oracles. We choose tasks with cheap electrical oracles. USB insertion is verified by device enumeration, and charger insertion by current draw. These oracles are used only for evaluation and for the one-bit episode label, never as a dense signal.

Simulation study. To study hacking at many random seeds cheaply, we also run a simulated version of the same question. We use ThreeDWorld's PyImpact contact sounds in a slot-drop task, with a detector trained on clean simulated clicks. This measures how quickly and in what ways precision collapses without harvesting.

Baselines

Sparse-reward reference. The do-nothing reference is residual RL with the sparse episode-end success bit as the only reward, in the style of HIL-SERL. It uses the same one bit of supervision we use, so any gain must come from how the audio localizes the event within the episode. The comparison point is the FTC paper's report that HIL-SERL averaged over 56 minutes of training and failed to converge on hard USB insertion even after two hours (FTC).

Strongest existing method. The strongest competitor is FTC's visual keyframe reward with its ACT base policy and residual MLP policy. It is strongest because it already solves this task suite. The paper reports a 100% success rate across all six tasks, an average of 27.5 minutes of training, convergence on hard USB insertion in 45 minutes, and 86.3% success at out-of-distribution positions (FTC).

We run FTC from the same demonstrations, on the same arm, with the same residual architecture and the same wall-clock budget. Hyperparameter tuning trials are matched across methods. In nominal lighting we do not expect to beat 100% success. Beating FTC means one of two things. We could match its success while converging faster on the non-monotonic hard USB task. Or we could retain substantially higher success under the glare, occlusion, and background-swap conditions where the FTC authors themselves report degradation. Hearing Touch observed a 60% versus 20% success drop between vision-only and audio-visual imitation policies under visual distractors (Hearing Touch). A gap of similar direction would count.

Force-torque threshold reward. A force-torque threshold reward, in the spirit of the hand-engineered force penalties in MSDP, provides the non-audio contact alternative. There is no reported result on Lacuna for force-threshold rewards on these insertion tasks, so we run it ourselves under the same budget.

Key ablation. The key ablation is an acoustic reward with exactly our detector and encoder, trained with the conventional sound-event-detection recipe: real clicks mixed onto the ambient bank only. A second ablation adds the ego-noise bank but omits harvesting. These isolate the harvested hard negatives as the ingredient that matters.

Signal-energy detector. A plain signal-energy threshold detector is included to show whether a learned detector is needed at all.

Why not just…?

Why not just use a force-torque sensor threshold as the reward? A force spike marks contact, not correct seating. A jammed plug and a seated plug can both produce high axial force, whereas the seating click is a distinct transient. Hearing Touch argues that audio also inherits internet-scale pretraining that touch lacks. In MSDP, force signals needed an asymmetric actor-critic design to keep training stable (MSDP). Still, on stiff insertions a well-tuned force rule may do fine, so we run it as a full baseline rather than dismiss it. If it matches us on precision under optimization, the audio contribution shrinks to cost and sensor convenience, and we will say so.

Why not just record more real clicks and train a bigger detector? More positives do not help with the failure mode we target, which is false positives on sounds the policy invents after the data was collected. Rare-event detection work shows that background diversity, not positive count, is what prevents shortcut learning (Rare Sound Event Detection). The ablation holding positives fixed while removing harvesting tests this directly. We additionally run a version with three times as many real positives but no harvesting. If that version keeps precision flat over training, our claim is wrong.

Why not just use the episode-end oracle as a sparse reward and skip audio? This is the HIL-SERL-style baseline above, and prior evidence says it is slow on tight insertions. It also cannot run where no electrical oracle exists. The detector, once trained, can. It is a genuine competitor on easy tasks, and we report where it ties.

Why not just denoise the audio before detection? A denoising front-end like the self-supervised network of Self-Supervised Voice Denoising Network removes stationary noise. Policy-generated tapping and scraping, however, are not noise in that sense: they are transient contact sounds, like the target. We test a denoiser-plus-conventional-detector variant. We predict it improves robustness to ambient noise but not to exploitation.

Proposed Experiments

Tasks and conditions. We use the FTC task suite, focusing on hard USB insertion, charger grasp-insertion, and card insertion. Each is run under three conditions: nominal lab; loud ambient playback plus a second robot running nearby; and visual perturbation (glare lamp, occluding cable, swapped tabletop). Metrics are oracle success rate, wall-clock time to 90% success, and reward precision under optimization tracked across training.

Decisive check 1: exploitation. Plot reward precision over training for the conventional-augmentation detector versus ours. The hypothesis requires the former to decline measurably while ours stays near its initial value. We also require that the declining runs show identifiable exploit behaviors, such as tapping or motor-speed patterns, in logged audio and video.

Decisive check 2: robustness. Under visual perturbation, the combined reward should retain markedly more success than FTC, while matching it in nominal conditions.

Decisive check 3: contact gating. On hard USB insertion, contact-gated rewards should shorten convergence relative to FTC's reported 45 minutes by permitting pull-back realignment.

Simulation arm. The ThreeDWorld study repeats check 1 across many seeds.

Modal-synthesis arm. This arm reports how far randomized synthesized clicks get toward real-exemplar performance. Given the ObjectFolder sim-to-real findings, we expect a gap, and a negative result there is informative rather than fatal.