Open-Set Ego-Noise Separation for Legged-Robot Audition via Annotation-Free Adaptation and Pretrained-Model Transfer

Koki Shoda, Jun Younes Louhi Kasahara, Aoba Koyanagi, Qi An, Atsushi YamashitaarXiv 2026 · arXiv

TL;DR

This paper introduces an open-set ego-noise separation framework that allows legged robots to filter out their own walking and motor sounds without requiring manually labeled data or quiet-room calibration. By combining an unsupervised graph-based noise discovery method (RecurGraph) with parameter-efficient transfer learning on a pretrained Diffusion Transformer (Transfer-DiT), the system effectively isolates arbitrary, unforeseen environmental sounds to enhance robotic audition.

What Problem Does This Paper Address?

Legged robots (like bipedal and quadrupedal platforms) are highly capable of navigating obstacles and stairs, making them ideal for autonomous security patrols. However, their onboard microphones are severely contaminated by "ego-noise"—the acoustic clutter of footstep impacts, joint-backlash rattling, and motor whines.

Existing audio separation methods typically suffer from two major flaws:

  1. High Adaptation Burden: Dictionary-based methods require clean, robot-specific ego-noise recorded in isolated, quiet environments for every combination of gait, speed, and flooring material.
  2. Closed-Set Limitations: Most deep-learning audio enhancement models are trained to extract specific classes (like human speech), failing to preserve arbitrary, unmodeled environmental sounds crucial for security, such as glass breaking, gunshots, or sirens (an open-set problem).

This paper tackles the challenge of achieving high-fidelity, open-set ego-noise removal using only unlabeled recordings collected during a brief pre-deployment "adaptation walk."

Methods

The authors propose a two-stage framework consisting of two novel components:

1. RecurGraph (Unsupervised Noise Selection)

To train a neural separator without manual labels, the robot must first identify which parts of its pre-deployment recordings contain pure ego-noise. RecurGraph achieves this by exploiting a simple statistical truth: robot noise is constant and recurring, whereas environmental sounds are intermittent and diverse.

Fig. 2: Overview of the proposed RecurGraph. The inputs are audio clips obtained by dividing recordings collected during pre-deployment adaptation walks into several-second segments. The VAE encoder and transformer encoder map each audio clip to an embedding. Global-average pooling of the embeddings constructs the embedding centroid, and similarities to the embedding centroid are thresholded to obtain initial scores. In parallel, Principal Component Analysis (PCA) produces reduced embeddings for graph construction. Graph-based score propagation propagates the initial scores over the reduced embeddings to obtain ego-noise-dominant scores for all audio clips.
  • Embedding Centroid: Audio clips are mapped to a latent space using a pretrained audio-language model. Averaging these vectors creates a "centroid" that naturally reflects the dominant, recurring ego-noise.
  • Graph-Based Score Propagation: Clips are ranked by similarity to this centroid. The top and bottom fractions serve as positive (pure noise) and negative seeds. Principal Component Analysis (PCA) reduces dimensionality, and a neighborhood graph propagates these initial labels to resolve ambiguous clips in the middle of the distribution.

2. Transfer-DiT (Pretrained-Model Transfer)

Instead of training a neural network from scratch—which risks overfitting to specific sounds—the authors adapt a general-purpose, zero-shot Diffusion Transformer (DiT) separator.

Fig. 3: Overview of the latent-space Flow Matching procedure for ego-noise separation. A frozen VAE maps the mixture waveform to a latent representation. Transfer-DiT estimates the velocity field, and numerical integration produces an ego- noise-free latent that the frozen VAE decoder reconstructs as environmental sound.

The text-prompt conditioning of the original model is stripped out. To specialize the network to the target robot while preserving its broad acoustic priors, the authors freeze the DiT backbone and insert trainable Bottleneck Adapter Layers (BAL) and Low-Rank Adaptation (LoRA) modules. The model is trained via latent-space Flow Matching on synthetic mixtures created by combining the selected ego-noise clips with a diverse environmental sound corpus.

Datasets

  • Ego-Noise Recordings: Collected using Unitree G1 (bipedal) and Unitree Go1 (quadrupedal) robots across four distinct real-world environments (outdoor plaza, indoor atrium, building lobby, and corridor) spanning concrete, tile, wood, carpet, and vinyl floors.
  • FSD50K: A large-scale dataset of human-labeled sound events used to provide diverse environmental background sounds during the training mixture phase.
  • Evaluation Corpora: Environmental sounds from ESC-50, TUT Rare Sound Events 2017, and the Human Screaming Detection Dataset were used to test downstream Environmental Sound Classification (ESC) and anomaly detection performance.

Key Insights and Ideas

  • Statistical Asymmetry as a Supervisor: Global average pooling across an unlabeled timeline successfully isolates machine noise from environmental events because environmental sounds lack temporal and acoustic consistency across long deployments.
  • Preserving the Open-Set Prior: Training a deep neural network from scratch on limited robot data causes it to forget how to handle rare or complex environmental sounds. Freezing the backbone and using parameter-efficient fine-tuning (LoRA/BAL) retains the model's "zero-shot" capacity to preserve unseen sound classes.
  • Graph Geometry over Raw Distance: Relying solely on raw embedding distance to a centroid fails when environmental sounds frequently recur (e.g., a steady cooling fan or repetitive background hum). Propagating scores through a localized neighborhood graph provides a much cleaner separation of ambiguous audio clips.

Key Results

The framework was validated using both controlled reference-audio mixtures and physical loudspeaker playback experiments with walking robots.

  • Fidelity and Separation Quality: Transfer-DiT consistently achieved the highest semantic similarity (CLAP scores) and perceptual quality (SAJ scores) compared to zero-shot baselines (like CLAPSep) and fully supervised models trained from scratch (like Conformer or Conv-TasNet).
  • Downstream Task Boost: In low Signal-to-Noise Ratio (SNR) conditions (-10 dB), the proposed method significantly improved downstream anomaly detection. For the quadrupedal robot, it raised the Area Under the Precision-Recall Curve ($AUPRC_{\text{anom}}$) from a noisy baseline of 0.61 up to 0.77.
  • Efficiency: Thanks to transfer learning, Transfer-DiT converged significantly faster than training from scratch, requiring a fraction of the training epochs (e.g., 29 epochs vs 137 epochs for the bipedal platform).

Limitations and Future Work

The primary limitation of RecurGraph is its reliance on the assumption that the pre-deployment recording contains a high enough proportion of ego-noise-dominant clips. If a robot adapts in an environment with continuous, non-stop background noise (like a loud factory floor), the embedding centroid will become contaminated.

The authors note that future work will explore continual-learning techniques to allow periodic, offline adaptation cycles during routine deployments, helping the robot adjust to new flooring materials or mechanical wear without forgetting previously encountered acoustic environments.

Summary updated

Abstract

This paper proposes an open-set ego-noise separation framework for legged-robot audition via annotation-free adaptation and pretrained-model transfer. The framework removes robot-specific ego-noise while preserving environmental sounds whose classes are not specified in advance. Acoustic sensing provides cues about a robot's surroundings beyond the visual field, but walking-induced ego-noise from footstep impacts, joint-backlash rattling, and motor noise severely contaminates the recordings. The framework first uses RecurGraph to select ego-noise-dominant clips from the unlabeled recordings by aggregating clip embeddings into an embedding centroid and propagating scores over an audio-embedding graph. The selected clips are mixed with diverse environmental sounds from a large-scale sound-event dataset to provide paired mixture--target supervision for open-set separation. Transfer-DiT then adapts a general-purpose zero-shot neural separator to achieve high-fidelity open-set ego-noise separation for the target robot. Experiments with bipedal and quadrupedal robots show reliable clip selection and improvements in separation quality and downstream task performance over baseline separators. These results demonstrate the feasibility of annotation-free adaptation without separately recorded ego-noise-only data or manual clip-level annotations.
BibTeX
@misc{shoda2026open,
  author = {Koki Shoda and Jun Younes Louhi Kasahara and Aoba Koyanagi and Qi An and Atsushi Yamashita},
  title = {{Open-Set Ego-Noise Separation for Legged-Robot Audition via Annotation-Free Adaptation and Pretrained-Model Transfer}},
  year = {2026},
  eprint = {2609.07440},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.07440},
}
Figure