Contact microphones are microphones pressed against a solid, such as a piezo patch on the chest, an in-ear mic that picks up skull-conducted sound, or a MEMS mic sealed inside a robot's silicone fingertip. They hear what touches them and largely ignore the room. That makes them attractive for wearables and robot hands, but it also creates a specific problem for training models across many devices. Every device hears the world through its own mechanical "lens": skin thickness, strap pressure, shoe sole, silicone cure. In federated learning, where devices train locally and send only model updates to a server, this per-device lens is what makes naive averaging go wrong.
Our proposal starts from one physical fact. A contact signal is roughly the source vibration passed through a device-specific linear filter, so on a log-spectrogram the filter becomes a simple per-frequency offset added to every frame. We propose to federate the event model while each client keeps a tiny local "channel vector" of a few dozen numbers, one gain per frequency band. Each device estimates this vector without labels, using only forward passes of the shared model. We test two claims. The first is that this restores accuracy on rare events under device heterogeneity at almost no on-device cost. The second is that keeping the channel local makes the shared updates less informative about who the wearer is.
Core Claim
Write the observed signal at client $k$ as $x_k(t) = s(t) * h_k(t) + \epsilon_k(t)$, where $s$ is the source excitation (a heartbeat, a cough, a fabric rubbing), $h_k$ is the client's mechanical transfer path, and $\epsilon_k$ is sensor noise. In the log-magnitude spectral domain, ignoring noise, this becomes
$$\log|X_k(f,\tau)| = \log|S(f,\tau)| + c_k(f),$$
where $\tau$ indexes time frames and $c_k(f) = \log|H_k(f)|$ is a fixed per-band offset, the channel vector. The claim is that in contact acoustics the bulk of the difference between clients lives in $c_k$ rather than in the event model. The shared network should therefore see $\log|X_k| - \hat c_k$, and only $\hat c_k$ should be personalized.
The obvious alternative is plain cepstral mean normalization, which subtracts each client's average spectrum. That fails in exactly the regime that matters here. The client average contains the source average too. A client whose recordings are dominated by breathing would have the spectral signature of breathing partly subtracted away, and clients differ sharply in their event mix. We instead estimate $c_k$ as the average residual between observed frames and shared class-conditional source templates $\mu_y(f)$, weighted by the current model's posterior $p(y \mid \cdot)$:
$$\hat c_k(f) = \frac{\sum_\tau \sum_y p(y \mid x_{k,\tau} - \hat c_k)\,\big(\log|X_k(f,\tau)| - \mu_y(f)\big)}{\sum_\tau 1}.$$
This is an expectation-maximization loop that alternates between guessing the event and re-estimating the channel. It needs no labels and no backpropagation on the device. The templates $\mu_y$ are federated along with the network. Because $c_k$ is never sent, the server never sees the part of the signal that encodes the device-and-body path. This matters because EarGate shows that speech-free, sub-50 Hz bone-conducted in-ear signals identify people at 97.26% balanced accuracy. Contact audio is biometric even when it contains no speech.
Positioning
Several lines of work motivate the proposal. Systems such as SonicSense, the Minsound/Minsight fabric-recognition hand, the chest-worn body-sound monitor and DreamCatcher show that contact sensing rejects ambient noise and carries information that vision and IMUs miss. In the fabric study, audio alone reached 95.06% against 41.15% for vision alone. All of these systems train centrally, and all of them document the channel problem. EarGate reports degradation across footwear, and the body-sound monitor reports artifacts that depend on mounting.
The federated audio work, including federated self-supervised acoustic event classification, federated acoustic modeling for ASR and a federated approach to acoustic models, handles airborne microphones. In that setting device heterogeneity is a minor nuisance compared with speaker and content variation. None of these papers models a per-client physical channel.
FedHealth supplies our closest method. It freezes shared convolutional layers, personalizes the fully connected layers, and aligns second-order feature statistics with a correlation-alignment (CORAL) loss on 50 Hz inertial data. We differ in what is personalized and where. FedHealth adapts a learned head with labels. We adapt a physically interpretable input-side offset without labels. We also treat the locality of that offset as a privacy property rather than only an accuracy device.
Research Program
We would build the method on two backbones: a small convolutional network on log-mel spectrograms that is plausible for a microcontroller, and a BEATs-style audio transformer of the kind used in DreamCatcher. In both, the input is channel-corrected by the local $\hat c_k$, and the class templates $\mu_y$ live in the same log-mel space and are averaged by the server alongside the weights. Training proceeds in alternating steps. A client runs a few expectation-maximization iterations to refresh $\hat c_k$ from its unlabeled stream. Clients that hold labels then take ordinary gradient steps on the corrected input. A cold-start client that holds no labels runs only the channel loop, which costs a handful of forward passes per minute of audio.
Three analyses accompany the method. The first is a channel-severity sweep. We re-filter real recordings with synthetic $h_k$ of increasing spectral tilt and notch depth, and with time-varying pressure-like drift, to find where the linear-offset assumption breaks. The second is a nonlinearity check on segments with clipping and motion shock, the failure mode the body-sound paper reports during running, to see whether the offset model degrades gracefully. The third is a leakage analysis. We train an attacker to re-identify subjects from the shared model's embeddings and from per-round updates, both with $c_k$ kept local and with it uploaded.
Baselines
The naive reference points are local-only training, in which each client trains on its own data, and plain federated averaging (FedAvg) on raw log-mel input. FedAvg is the standard aggregation scheme described in Decentralized Model Training via Federated Learning. These two runs bound the problem. If FedAvg already matches pooled training, the heterogeneity is not severe and the proposal has no target.
The centralized upper reference is the same backbone trained on pooled data. On DreamCatcher, the cited paper reports wearer-aware sleep sound event classification accuracy of "approximately 90%" with BEATs. That is the strongest reported number on our primary benchmark. We expect any federated method to fall short of it, and we will report the size of the gap.
The strongest existing federated competitor is FedHealth-style personalization: a shared feature extractor, a locally fine-tuned head, and CORAL alignment, reimplemented on our backbones. FedHealth reports 99.4% average accuracy on UCI smartphone activity recognition and a 5.3% absolute gain over its non-federated baseline. However, I found no reported result on Lacuna for FedHealth or any federated method on contact-acoustic data, so all federated comparisons here are new runs. All federated methods share the same client partitions, number of rounds, local epochs, and hyperparameter-search budget, and FedHealth receives the same labeled fraction per client that our method uses.
Two training-free front-ends are run with FedAvg: per-client cepstral mean normalization and per-channel energy normalization. They isolate whether the learned, label-aware channel estimate matters or whether any channel removal suffices. A further baseline keeps batch-normalization statistics local, an implicit per-client affine correction. I found no paper in the research file describing it, so it is included as a widely used design rather than a cited one.
The key ablations swap the channel estimator while keeping everything else fixed. The variants are the full expectation-maximization estimate, an oracle estimate computed from labels, and mean subtraction. A separate ablation uploads $c_k$ to the server instead of keeping it local, which isolates the privacy effect.
Beating the field means three things. On rare-class macro-F1 and worst-decile per-client accuracy, the method should exceed both FedHealth and cepstral mean normalization with FedAvg, with confidence intervals across random client splits that do not overlap. It should do this while sending fewer personalized parameters. And it should lower subject re-identification from shared embeddings relative to FedAvg.
Why not just…?
Why not just subtract each client's mean spectrum? This is the most serious threat, because under our own model it is nearly the right answer when every client has the same mix of events. It goes wrong when the event mix differs, because the mean then absorbs source content. DreamCatcher's skew of more than 137,000 breathing events against 95 coughs means client mixes will differ enormously. The ablation settles this. We will vary label skew across clients and plot both methods. If mean subtraction matches ours even under strong skew, we will report that cheap normalization suffices and drop the estimator.
Why not just personalize the classifier head as FedHealth does? A personalized head needs local labels, and labels for rare contact events are the scarcest thing at the edge. FedHealth's own design assumes labeled target subjects. Head personalization also corrects the channel only indirectly, after the shared filters have already been trained on mixed channels. The comparison is the cold-start experiment. New clients receive zero labels and we measure accuracy after $k$ unlabeled minutes. FedHealth cannot adapt at all in that regime, while our estimator can.
Why not just train one central model with random-equalizer augmentation and deploy it frozen? This is a real competitor for accuracy if raw data can be pooled. The EarGate result argues that such data is biometric, which limits pooling. Augmentation can also be added to any federated method, so we will run it both alone and combined with ours. If augmentation alone matches ours, the remaining case for the proposal rests on the privacy and cold-start results. We will say so plainly.
Proposed Experiments
The primary benchmark is DreamCatcher. Each of its 24 subjects is a client, the task is classifying sleep events from the inward-facing feedback microphone, and we report results with and without the IMU channel. Natural partner pairs provide realistic shared-environment structure. The second benchmark is EarGate's footwear-varied gait recordings from 31 subjects, if the data can be obtained. In that setting a client is a subject-and-shoe combination, which gives a clean test of whether shoe changes are absorbed by $c_k$.
For robotics, no multi-device contact-acoustic dataset exists. We would build a small fleet of SonicSense-style hands, which the paper reports at $215 per unit, and fit them with fingertips that deliberately vary in silicone thickness and mic placement. Each fingertip is a client. Tasks are material classification, scored with the SonicSense F1 protocol, and fabric recognition, following the Minsound rubbing protocol. This directly tests channel heterogeneity across hardware of the kind a deployed fleet would have.
Metrics are rare-class macro-F1 (cough, teeth grinding), worst-decile client accuracy, cold-start accuracy as a function of unlabeled minutes, bytes sent per round, on-device energy per adaptation step, and subject re-identification accuracy of an attacker reading shared embeddings.
The decisive checks are three. The learned $\hat c_k$ should correlate with independently measured device responses, from tap-test sweeps on the robot fingertips and swaps between shoe conditions. Cold-start clients should recover most of the gap to the label-oracle channel. Keeping $c_k$ local should cut re-identification relative to uploading it. If re-identification is unchanged, the identity information sits in the source dynamics rather than the channel, and we would retract the privacy half of the claim.
References
- EarGate: gait-based user identification with in-ear microphones
- SonicSense: Object Perception from In-Hand Acoustic Vibration
- Adding Internal Audio Sensing to Internal Vision Enables Human-Like In-Hand Fabric Recognition with Soft Robotic Fingertips
- A First Step Towards On-Device Monitoring of Body Sounds in the Wild
- DreamCatcher: A Wearer-aware Multi-modal Sleep Event Dataset Based on Earables in Non-restrictive Environments
- Federated Self-Supervised Learning for Acoustic Event Classification
- Federated Acoustic Modeling for Automatic Speech Recognition
- A Federated Approach in Training Acoustic Models
- FedHealth: A Federated Transfer Learning Framework for Wearable Healthcare
- Introduction to the Minitrack on Decentralized Federated Learning: Applications, Solutions, and Challenges