Letting the Accelerometer Decide Which Microphone Errors Matter

Microphones are cheap, contact-free sensors for monitoring rails, bridges, and machines, but in the field their faint damage signatures are buried under wind, traffic, and engine noise. Real damage is also rare, so the usual fix is focal loss, a training objective that down-weights examples the model already gets right and concentrates learning on the ones it gets wrong. In noisy acoustic data this backfires. The examples a microphone classifier gets most wrong are often not hard damage cases. They are noise bursts, mislabeled windows, or temperature effects that only look like damage. Focal loss pours gradient into exactly these cases.

We propose to use a contact accelerometer, which is largely immune to airborne noise, as a training-time referee. When the microphone model fails on a window, the vibration model's confidence on that same window tells us why it failed. If the vibration channel recognizes the true class, the damage signature is physically present and the acoustic model should be pushed hard to find it. If the vibration channel is also lost, the window is more likely noise, a bad label, or genuine ambiguity, and it should not dominate training. The accelerometer is removed at deployment, so only the microphone is used in the field. The method is a cross-modal focal weight plus an attention-transfer term, added to an acoustic classifier that works on image encodings of the signal.

Core Claim

Under severe class imbalance and low signal-to-noise ratio (SNR), deciding how hard to train on an acoustic example should depend on a noise-immune second sensor's confidence, not only on the acoustic model's own confidence. We claim this improves rare-damage recall at a fixed false-alarm rate and improves calibration compared with standard focal loss, class weighting, and plain distillation from a vibration teacher. The gain should grow as SNR falls and as label noise rises.

For acoustic window $i$ with true class $y_i$, let $p^{a}_i$ be the acoustic model's probability for $y_i$. Let $\hat p^{v}_i$ be the vibration teacher's probability for $y_i$, computed out-of-fold so the teacher has never trained on window $i$. The training loss is

$$ \mathcal{L} = -\sum_i \alpha_{y_i}\,(1-p^{a}_i)^{\gamma}\,(\hat p^{v}_i)^{\beta}\,\log p^{a}_i \;+\; \lambda \sum_i \hat p^{v}_i\,\mathrm{KL}\!\left(A^{v\to a}_i \,\|\, A^{a}_i\right). $$

In the first term, $\gamma$ is the usual focusing exponent, $\alpha_{y}$ is a class prior weight, and $\beta \ge 0$ sets how strongly the vibration channel vetoes examples. At $\beta=0$ the loss reduces to standard focal loss. In the second term, $A^{v\to a}_i$ is the attention distribution over acoustic time patches produced by a training-only cross-attention block, in which vibration features ask "where in the acoustic signal is the structural event?". $A^{a}_i$ is the acoustic student's own attention over the same patches, and $\lambda$ weights this term. The distillation is weighted by $\hat p^{v}_i$ for the same reason as the first term: only trust the teacher's map where the teacher is right.

Relation to Prior Work

Two papers supply the core ingredients. Cross Modal Focal Loss for RGBD Face Anti-Spoofing modulates one camera branch's loss using the other branch's confidence in a two-stream RGB-depth model. Both streams are present at test time, and the goal is to stop each branch from relearning what the other already knows. We invert the purpose. The second sensor is privileged: available only during training. Its confidence acts as a reliability score for the deployed sensor's hard examples, which turns the focal weight from "redundancy suppression" into "noise versus signal triage."

Acceleration-guided Acoustic Signal Denoising, known as AG-ASDF, already uses clean track acceleration to supervise a learnable wavelet denoiser for microphones and then drops the accelerometer at inference. It reaches 95.4% accuracy on three slab-track support conditions. This is the closest existing system and the one we must beat. It transfers the accelerometer's knowledge through signal reconstruction, uses a standard classifier with no imbalance handling, and treats every training pass as equally trustworthy. We transfer the knowledge through the loss weighting and the attention, and we test under imbalance and label corruption, which it does not.

Several papers motivate the problem rather than supply a method. P2IFormer (P2IFormer) encodes bearing vibration as Gramian Angular Fields and remains accurate down to -6 dB on balanced data, but explicitly leaves imbalanced fault data as future work. The SqueezeNet attention study reports that imbalance on ToyADMOS hurt convergence.

Two lines of work constrain the design. Revisiting Reweighted Risk for Calibration and Dual Focal Loss for Calibration show that focal reweighting distorts confidence and over-weights corrupted samples. That is precisely the failure we target, so calibration is a primary metric rather than an afterthought. Work on temperature and humidity effects on concrete (Impact of temperature and relative humidity variations on coda waves in concrete) warns that environmental shifts can fool contact sensors too. Our veto therefore risks discarding genuinely hard damage that both sensors miss, and we measure this directly.

The image encoding is a supporting choice. A Gramian Angular Difference Field (GADF) turns a length-$N$ window into an $N\times N$ image whose entry $(i,j)$ is $\sin(\phi_i-\phi_j)$, where $\phi=\arccos$ of the rescaled signal. P2IFormer found GADF beat the summation variant. We use GADF patches for both sensors so their time indices align patch-for-patch for the cross-attention block.

GADF has a known weakness: its min-max rescaling lets one noise spike squash the whole signal into a narrow band. We therefore rescale using clipped quantiles instead. We do not assume GADF beats spectrograms. A log-mel front end is a planned ablation, because the AG-ASDF paper argues that fixed-window spectrograms struggle with moving-train transients.

Research Program

The first stage trains a vibration teacher on accelerometer GADF images with class-weighted cross-entropy using five-fold cross-fitting. Every training window then receives an out-of-fold confidence $\hat p^{v}_i$. Without cross-fitting, an overfit teacher would report confidence near 1 everywhere and the veto would do nothing.

The second stage trains a fused model. Vibration patch tokens query acoustic patch tokens through a gated cross-attention block, following the gating design of Gated Cross Attention, which mixes the attended context with the original features through a learned gate. The attention rows are extracted as $A^{v\to a}$. This fused model is never deployed.

The third stage trains the acoustic-only student with the loss above. We sweep $\beta \in \{0, 0.5, 1, 2\}$ and $\lambda$ on a validation split. All analyses bin the training examples into four quadrants: both models right, acoustic-only wrong, vibration-only wrong, and both wrong. We then report where gradient mass actually goes under each loss. This shows whether the mechanism works for the stated reason.

Baselines

The naive reference point is an acoustic-only GADF classifier trained with plain cross-entropy on the imbalanced data. Every other method must clear it, and it shows how badly imbalance and noise interact.

The next tier holds the network fixed and changes only the objective. The objectives are class-weighted cross-entropy, standard focal loss (Focal Loss for Dense Object Detection), and the calibration-aware variants AdaFocal and Dual Focal Loss. These are the strongest competitors on the imbalance side, because they address focal loss's calibration failure without a second sensor. If one matches us, the accelerometer adds nothing to the weighting.

The strongest existing method on the cross-modal side is AG-ASDF. We reimplement it on the same splits and follow it with the same classifier head, so the only difference is denoise-then-classify versus reweight-and-distill. Its reported figure is 95.4% classification accuracy on the slab-track dataset under balanced classes. No reported result was found on Lacuna for AG-ASDF or any acoustic SHM method under induced class imbalance.

On the motor benchmarks, the best reported figures are those of MM-HCAN: 99.61% accuracy on bearing faults and under 1.5% drop at 10 dB noise, with both modalities used at test time. We run a fused model in the MM-HCAN style as an upper bound that keeps both sensors at test, not as a competitor.

Two ablations isolate the key ingredient. In the first, $\hat p^{v}$ is replaced by the acoustic model's own out-of-fold confidence from an ensemble, which tests whether the second modality matters or any outside confidence estimate would do. In the second, $\hat p^{v}$ is randomly permuted across examples, which controls for the overall weight distribution. We also run $\lambda=0$ to separate the loss weighting from the attention transfer, and GADF versus log-mel to separate the representation from the objective.

All methods share the same backbone, data, epochs, and a fixed budget of 30 hyperparameter trials each, so tuning effort is equal. Beating a baseline means higher minority-class recall at a 1% false-alarm rate, with non-overlapping 95% bootstrap intervals over five seeds, and no worse expected calibration error.

Why not just…?

Why not just distill the vibration teacher into the acoustic student with soft labels? This is the most serious threat. Soft-label distillation already passes the teacher's confidence along, and it may implicitly soften noisy examples. The difference is that distillation copies the teacher's uncertainty as a target. On windows where both sensors fail, it still asks the student to fit a confused distribution, and it does nothing to stop focal weighting from amplifying those windows. We run temperature-tuned distillation, alone and combined with focal loss, as a baseline. If it matches us in the "both wrong" quadrant analysis and on minority recall, we conclude the veto is unnecessary and say so.

Why not just denoise the microphone first, as AG-ASDF does? Denoising removes airborne noise but leaves label noise and environmental mimicry untouched. It also does nothing about the 99-to-1 imbalance, since the AG-ASDF evaluation used balanced passes. Denoising and our loss are also compatible. If AG-ASDF plus focal loss closes the gap, we will report the combination as the practical recommendation.

Why not just use class weights or standard focal loss? The calibration literature cited above documents that focal reweighting amplifies corrupted samples, and nothing in a single-sensor loss can tell a noise burst from a masked crack. The $\beta=0$ setting of our loss is exactly standard focal loss. The $\beta$ sweep therefore directly measures what the vibration channel adds.

Why not just keep the accelerometer in the field? Where contact sensors are affordable, the fused upper bound is the right answer. The motivation here is sites where mounting and cabling dense contact arrays on rail tracks or turbine blades is impractical, the deployment constraint stressed by AG-ASDF. The upper bound tells practitioners how much accuracy dropping the accelerometer costs.

Proposed Experiments

The primary testbed is the slab-track dataset from AG-ASDF: paired 20 kHz accelerometer and microphone recordings over mortar, rubber, and spring supports at 20 to 80 km/h. Its public availability is not stated in the file, so access must be confirmed with the authors.

The secondary testbed is the induction-motor benchmarks used by MM-HCAN. There, motor current plays the cheap deployed sensor and vibration plays the privileged one, which tests whether the idea transfers beyond acoustics.

On both testbeds we impose imbalance by subsampling damaged classes to 1% and 5% of training windows, keeping test sets with natural ratios reported separately. We mix in real industrial noise drawn from the normal recordings of the wood planer dataset at SNRs from +10 dB to -10 dB. We also corrupt 0%, 5%, or 15% of labels, half by random flips and half by relabeling noise-burst windows as damage, to mimic false events.

The metrics are minority recall at 1% false alarm, macro-F1, expected calibration error, and area under the risk-coverage curve. We additionally report the share of gradient mass in each of the four agreement quadrants.

The decisive checks are three. First, the advantage of the full loss over standard focal loss should widen monotonically as SNR drops and as label corruption rises. Second, the permuted-confidence and self-confidence ablations should recover little of that advantage, which shows the second modality is what matters. Third, among genuinely damaged windows that both sensors misclassify, recall should not fall by more than a small, pre-registered margin relative to focal loss. If it does, the veto is discarding real hard damage, and the method needs a floor on $\hat p^{v}$ rather than a pure power law.