Picture a meeting recorded by several everyday devices at once: phones, earbuds, a laptop. Each device has its own cheap quartz clock, so the clocks slowly disagree. Today, multi-device transcription works by sending raw audio from every device to a central computer, which then corrects the timing differences at the level of individual audio samples. That design is expensive for small devices, and it exposes every participant's voice, a biometric, to whatever computer receives it. This note proposes the reverse. Each device runs a small causal encoder, meaning it uses only past and present audio and never waits for the future. The encoder turns speech into a slow stream of features that describe what is being said but not who is saying it. The central computer then corrects for clock drift on those feature streams, not on the waveforms.
The bet rests on one observation. Clock drift and voice identity live in different parts of the signal. Drift can be estimated from when sounds happen, and every device hears the same words at almost the same moments. Identity lives in timbre and pitch, which the task does not need. If that holds, a feature stream stripped of identity is still a good signal for estimating drift. Working at a frame rate of about 100 Hz instead of 16 kHz also makes drift tracking roughly 160 times cheaper and pushes all of it off the edge device.
Core Claim
A device clock that runs fast or slow by a fraction $\epsilon_k$ (for example $\epsilon_k = 10^{-4}$ for 100 parts per million) and starts with offset $\delta_k$ maps true time $t$ to device time $(1+\epsilon_k)t + \delta_k$. After $T$ seconds, the misalignment between device $k$ and a reference, measured in feature frames of hop $h$, is
$$ \Delta_k(T) = \frac{\epsilon_k T + \delta_k}{h}. $$
With $h = 10$ ms, 50 ppm over a 10-minute session gives 3 frames, and 100 ppm over an hour gives 36 frames. This sets the problem's scale in both directions.
On one side, drift at feature rate is small enough per frame that waveform phase never needs to be recovered, and the hub never needs raw audio. On the other, the misalignment keeps growing. A recurrent network trained with drift augmentation will absorb it for a while and then lose track. This matches the long-session "slippage" failure noted for end-to-end drift absorption in Elucidating Clock-drift Using Real-world Audios….
We claim that three components together give distant meeting transcription where no raw audio leaves any device:
- A causal, identity-scrubbed feature encoder on each device.
- A two-number-per-device offset-and-skew tracker on the hub.
- Causal cubic interpolation along the feature time axis.
We expect this system to stay close to raw-audio systems in word error rate and to stay flat across drift rates and session lengths. A speaker-verification attacker working on the transmitted features should be pushed toward chance.
Relation to Prior Work
Several papers motivate the gap. Cooperative Speech Separation With a Microphone Array and Asynchronous Wearable Devices is the closest existing work. Its wearables send scalar speech-presence probabilities, a per-frame guess of whether speech is present, instead of audio. This sidesteps drift by never combining phase across devices. Its features were chosen for drift robustness, not privacy, and the paper reports that the probabilities degrade at high frequencies where neighbours' speech leaks in. We keep its central move of sharing only low-rate features, but replace the hand-designed scalar with a learned content stream trained against identity attackers. We also add explicit alignment so the features can be combined across devices frame by frame.
End-to-end Streaming model for Low-Latency Speech Anonymization shows that a causal content encoder with a stop-gradient against speaker information fits a small budget: its Lite variant is about 10 MB with 66 ms CPU latency at 40 ms chunks. It is single-channel and assumes a perfect clock. We reuse its content-encoder recipe as our edge model, but we never resynthesize audio.
On the drift side, the methods are either heavy or rely on lookahead. Clock Skew Robust Acoustic Echo Cancellation tracks skew with a subband Kalman filter whose hop had to shrink from 160 to 40 samples. That is costly, and it works on raw subbands. Cross-Talk Reduction absorbs jitter with non-causal filter taps and reports that larger drift breaks it. Our tracker is the same kind of state-space model, but it runs on two scalars per device at frame rate, which is why it is cheap.
The method for the interpolation step comes from High-quality causal interpolation for online unidimensional signal processing. It gives a strictly causal cubic pre-filter costing one halving and one addition per sample. We apply it per feature dimension on the hub, at the cost of a fixed delay of about two frames.
The ad hoc separation systems constrain what the hub must handle. Continuous Speech Separation with Ad Hoc Microphone Arrays shows that speech gets duplicated across devices unless active speakers are counted. Enhanced Deep Speech Separation in Clustered Ad Hoc Distributed Microphone Environments shows that choosing a reference device matters. Attention-based distributed speech enhancement for unconstrained microphone arrays with varying number of nodes shows how to tolerate devices dropping out. All three send raw audio and use non-causal context.
The privacy literature is about images. Deep Private-Feature Extraction and PATROL remove identity with contrastive or adversarial losses and inversion-resistant training. DistPrivacy splits feature maps across untrusted devices so no single device holds a complete set. We borrow PATROL-style adversarial inversion training. None of these papers deals with streams whose time axes disagree.
Research Program
Edge encoder. Each device runs a causal convolutional encoder sized to the Lite anonymizer's budget or smaller. It emits a 10 ms-hop vector made of two parts. The first is content features, close to self-supervised phonetic units. The second is a few band-energy channels that carry a coarse picture of how loud the room is around that device.
The encoder is trained against three objectives:
- the downstream transcription loss;
- a speaker-verification adversary and a waveform-inversion adversary, both trained against it in a min-max loop;
- a timing-preservation loss, a contrastive objective that asks a small head to predict the relative lag between two devices' streams of the same audio.
The timing loss is the new ingredient. Without it, the adversarial pressure is free to blur the sharp onsets that drift estimation needs.
Hub aligner. For each device, the hub cross-correlates the device's content stream with the reference device over a sliding window of past frames. It picks the reference by coherence, as in the clustered ad hoc work. The resulting lag estimates feed a two-state Kalman filter holding the offset $\delta_k$ and the skew $\epsilon_k$. The filter's forecast lag drives causal cubic resampling of that device's feature stream onto the reference timeline.
Hub recognizer. The aligned streams are fused by attention across devices, which tolerates devices dropping out, and passed through a speaker-counting head before a streaming recognizer. Speaker labels for diarization are anonymous and local to the session. They come from which devices hear a stream loudly, not from voiceprints. As a result, the system can say "speaker A spoke here" within one meeting but cannot link that speaker to an enrolled identity.
Analysis question. The key scientific question is whether privacy and alignability can coexist. We will trace a curve with drift-estimation error on one axis and attacker equal error rate on the other, by sweeping the strength of the adversarial loss with and without the timing loss.
Baselines
The naive reference point sends raw 16 kHz audio from every device to the same hub recognizer, with no drift handling. A second naive point uses only the single best device. Together they bracket what can be achieved with no privacy at all and what the multi-device setup adds.
The strongest existing utility system is the ad hoc continuous speech separation pipeline of Continuous Speech Separation with Ad Hoc Microphone Arrays. It is non-causal, sends raw audio, and uses speaker counting. On AdHoc-LibriCSS at 20–30% overlap it reports a word error rate of 15.17%, down from 25.08% unprocessed. On single-talker segments it reports 11.67% with speaker counting. We will rerun it with the same recognizer backend as our system and treat its numbers as a ceiling, not a target. Coming within a stated margin of it while staying causal and private would count as success.
On CHiME-7 DASR, the CHiME-7 challenge baseline reports a macro diarization-attributed word error rate (DA-WER) of 55.3% with automatic diarization and 33.4% with oracle diarization. CTRnet reports 22.6% against 26.6% for guided source separation, in the condition described in Cross-Talk Reduction. We will compare against the official baseline under matched automatic diarization. Beating 55.3% with transmitted features only would be a meaningful result.
The strongest privacy competitor runs the Lite streaming anonymizer of End-to-end Streaming model for Low-Latency Speech Anonymization on each device. Each device sends anonymized audio to a drift-augmented hub separator. That paper reports a word error rate of 5.12% for its Base model on LibriTTS, and an equal error rate "comparable to heavy transformer-based baselines." It reports no multi-device number. We beat this baseline if we match its attacker equal error rate while achieving lower DA-WER, at no more edge compute.
The speech-presence-probability baseline from Cooperative Speech Separation… has no reported transcription number in the file; there is no reported result found on Lacuna for it on these benchmarks. We will reimplement it, sending per-device probabilities into the same hub.
Ablations isolate each ingredient: the aligner removed but drift augmentation kept; the timing-preservation loss removed; the adversarial losses removed; and the feature-domain aligner replaced by a sample-domain resampler on the device, for the compute comparison.
Fairness rules apply throughout. All systems share the same training corpora (LibriTTS and VoxCeleb for identity, plus simulated ad hoc rooms) and the same recognizer. All get the same hyperparameter-search budget. Edge-side compute and memory are matched to the Lite anonymizer's footprint.
Why not just…?
Why not just synchronize clocks with the network time protocol (NTP) or the precision time protocol (PTP)? At feature rate this is a real threat. The 10–50 ms network time protocol errors cited in Mitigating Clock Drift in Distributed Sensor Networks are 1–5 frames. That is not negligible, but it is far less damaging than at sample rate. Two further problems remain. Timestamps reflect the system clock rather than the clock driving the audio converter. And Bluetooth buffering adds fluctuating delay, as measured in Elucidating Clock-drift…. We will run a baseline that aligns streams by simulated timestamps with jitter matching those error levels. If it matches our aligner, we will report the aligner as unnecessary wherever timestamps exist.
Why not just share speech-presence probabilities? They are private by construction and immune to drift. However, they carry no phonetic content, so they cannot help recognize overlapped speech. Their own authors report degradation at high frequencies from neighbouring talkers. The clustered ad hoc work also found that coarse masking underperforms reference-based fusion. The speech-presence baseline above tests this directly on overlap-stratified word error rate.
Why not just anonymize audio on each device and send it to a standard separator? Each device would resynthesize the meeting with its own pseudo-voice and its own processing delay. That destroys the cross-device consistency the separator relies on. It also spends edge compute on a decoder whose output only gets re-encoded at the hub. This is the main privacy baseline, so the comparison is run rather than argued.
Why not just train with drift augmentation and skip explicit tracking? The equation above says the misalignment grows linearly with session length. We therefore expect augmentation alone to hold for short clips and fail on hour-long sessions. The ablation sweeps drift from 0 to 200 ppm and session length from 1 to 60 minutes. If augmentation alone stays flat, the tracker is dropped from the claim.
Why not use homomorphic encryption? Encrypted computation cannot meet a sub-100 ms streaming budget on wearables, per Recent Advances in Mobile Middleware…. We do not evaluate it.
Proposed Experiments
The main evaluations run on AdHoc-LibriCSS (5 real asynchronous devices) and CHiME-7 DASR (real multi-device meetings). Controlled sweeps use simulated ad hoc rooms built from LibriSpeech, with drift injected by resampling each device by $\epsilon_k \in [-200, 200]$ ppm and with random device dropouts.
Privacy is measured with two attackers trained on the transmitted features:
- a speaker-verification attacker, scored by equal error rate under both ignorant and informed threat models, where the informed attacker knows the encoder;
- an inversion network, whose reconstructions are scored for speaker equal error rate and intelligibility.
Utility is measured by word error rate stratified by overlap ratio, and by DA-WER. Cost is measured by edge latency, memory, and transmitted bits per second against raw audio.
Three outcomes decide the proposal:
- Drift robustness. With the aligner, word error rate stays flat across the drift and session-length sweep, while the no-aligner ablation degrades as $\Delta_k(T)$ grows.
- Privacy without losing alignment. With the timing loss, the attacker's equal error rate approaches chance while lag-estimation error stays under one frame. Without the timing loss, the same privacy strength raises lag error and word error rate.
- Trade-off. The full system beats both the per-device anonymizer baseline and the speech-presence baseline in DA-WER at matched privacy and edge compute, and stays within a pre-registered margin of the raw-audio, non-causal ceiling.
If the second outcome fails, meaning identity scrubbing inevitably blurs timing, that is itself a useful negative result. It would show that private multi-device systems need trusted timestamps.
References
- Elucidating Clock-drift Using Real-world Audios In Wireless Mode For Time-offset Insensitive End-to-End Asynchronous Acoustic Echo Cancellation
- Cooperative Speech Separation With a Microphone Array and Asynchronous Wearable Devices
- End-To-End Streaming Model For Low-Latency Speech Anonymization
- Clock Skew Robust Acoustic Echo Cancellation
- Cross-Talk Reduction
- High-quality causal interpolation for online unidimensional signal processing
- Continuous Speech Separation with Ad Hoc Microphone Arrays
- Enhanced Deep Speech Separation in Clustered Ad Hoc Distributed Microphone Environments
- Attention-based distributed speech enhancement for unconstrained microphone arrays with varying number of nodes
- Deep Private-Feature Extraction
- PATROL: Privacy-Oriented Pruning for Collaborative Inference Against Model Inversion Attacks
- DistPrivacy: Privacy-Aware Distributed Deep Neural Networks in IoT surveillance systems
- The CHiME-7 DASR Challenge: Distant Meeting Transcription with Multiple Devices in Diverse Scenarios
- A Novel Two-way Time Synchronization Protocol for Fusion Application
- Recent Advances in Mobile Middleware for Wireless Systems and Services