TL;DR
This paper introduces an open-set ego-noise separation framework that allows legged robots to filter out their own walking and motor sounds without requiring manually labeled data or quiet-room calibration. By combining an unsupervised graph-based noise discovery method (RecurGraph) with parameter-efficient transfer learning on a pretrained Diffusion Transformer (Transfer-DiT), the system effectively isolates arbitrary, unforeseen environmental sounds to enhance robotic audition.
What Problem Does This Paper Address?
Legged robots (like bipedal and quadrupedal platforms) are highly capable of navigating obstacles and stairs, making them ideal for autonomous security patrols. However, their onboard microphones are severely contaminated by "ego-noise"—the acoustic clutter of footstep impacts, joint-backlash rattling, and motor whines.
Existing audio separation methods typically suffer from two major flaws:
- High Adaptation Burden: Dictionary-based methods require clean, robot-specific ego-noise recorded in isolated, quiet environments for every combination of gait, speed, and flooring material.
- Closed-Set Limitations: Most deep-learning audio enhancement models are trained to extract specific classes (like human speech), failing to preserve arbitrary, unmodeled environmental sounds crucial for security, such as glass breaking, gunshots, or sirens (an open-set problem).
This paper tackles the challenge of achieving high-fidelity, open-set ego-noise removal using only unlabeled recordings collected during a brief pre-deployment "adaptation walk."
Methods
The authors propose a two-stage framework consisting of two novel components:
1. RecurGraph (Unsupervised Noise Selection)
To train a neural separator without manual labels, the robot must first identify which parts of its pre-deployment recordings contain pure ego-noise. RecurGraph achieves this by exploiting a simple statistical truth: robot noise is constant and recurring, whereas environmental sounds are intermittent and diverse.
- Embedding Centroid: Audio clips are mapped to a latent space using a pretrained audio-language model. Averaging these vectors creates a "centroid" that naturally reflects the dominant, recurring ego-noise.
- Graph-Based Score Propagation: Clips are ranked by similarity to this centroid. The top and bottom fractions serve as positive (pure noise) and negative seeds. Principal Component Analysis (PCA) reduces dimensionality, and a neighborhood graph propagates these initial labels to resolve ambiguous clips in the middle of the distribution.
2. Transfer-DiT (Pretrained-Model Transfer)
Instead of training a neural network from scratch—which risks overfitting to specific sounds—the authors adapt a general-purpose, zero-shot Diffusion Transformer (DiT) separator.
The text-prompt conditioning of the original model is stripped out. To specialize the network to the target robot while preserving its broad acoustic priors, the authors freeze the DiT backbone and insert trainable Bottleneck Adapter Layers (BAL) and Low-Rank Adaptation (LoRA) modules. The model is trained via latent-space Flow Matching on synthetic mixtures created by combining the selected ego-noise clips with a diverse environmental sound corpus.
Datasets
- Ego-Noise Recordings: Collected using Unitree G1 (bipedal) and Unitree Go1 (quadrupedal) robots across four distinct real-world environments (outdoor plaza, indoor atrium, building lobby, and corridor) spanning concrete, tile, wood, carpet, and vinyl floors.
- FSD50K: A large-scale dataset of human-labeled sound events used to provide diverse environmental background sounds during the training mixture phase.
- Evaluation Corpora: Environmental sounds from ESC-50, TUT Rare Sound Events 2017, and the Human Screaming Detection Dataset were used to test downstream Environmental Sound Classification (ESC) and anomaly detection performance.
Key Insights and Ideas
- Statistical Asymmetry as a Supervisor: Global average pooling across an unlabeled timeline successfully isolates machine noise from environmental events because environmental sounds lack temporal and acoustic consistency across long deployments.
- Preserving the Open-Set Prior: Training a deep neural network from scratch on limited robot data causes it to forget how to handle rare or complex environmental sounds. Freezing the backbone and using parameter-efficient fine-tuning (LoRA/BAL) retains the model's "zero-shot" capacity to preserve unseen sound classes.
- Graph Geometry over Raw Distance: Relying solely on raw embedding distance to a centroid fails when environmental sounds frequently recur (e.g., a steady cooling fan or repetitive background hum). Propagating scores through a localized neighborhood graph provides a much cleaner separation of ambiguous audio clips.
Key Results
The framework was validated using both controlled reference-audio mixtures and physical loudspeaker playback experiments with walking robots.
- Fidelity and Separation Quality: Transfer-DiT consistently achieved the highest semantic similarity (CLAP scores) and perceptual quality (SAJ scores) compared to zero-shot baselines (like CLAPSep) and fully supervised models trained from scratch (like Conformer or Conv-TasNet).
- Downstream Task Boost: In low Signal-to-Noise Ratio (SNR) conditions (-10 dB), the proposed method significantly improved downstream anomaly detection. For the quadrupedal robot, it raised the Area Under the Precision-Recall Curve ($AUPRC_{\text{anom}}$) from a noisy baseline of 0.61 up to 0.77.
- Efficiency: Thanks to transfer learning, Transfer-DiT converged significantly faster than training from scratch, requiring a fraction of the training epochs (e.g., 29 epochs vs 137 epochs for the bipedal platform).
Limitations and Future Work
The primary limitation of RecurGraph is its reliance on the assumption that the pre-deployment recording contains a high enough proportion of ego-noise-dominant clips. If a robot adapts in an environment with continuous, non-stop background noise (like a loud factory floor), the embedding centroid will become contaminated.
The authors note that future work will explore continual-learning techniques to allow periodic, offline adaptation cycles during routine deployments, helping the robot adjust to new flooring materials or mechanical wear without forgetting previously encountered acoustic environments.
Summary updated