When the Outlier Is the Diagnosis: Quantizing Lung-Sound Transformers Without Erasing Crackles

Transformer models that classify stethoscope recordings are now the most accurate automated lung-sound screeners. They are also too large and too dependent on floating-point arithmetic to run inside the battery-powered devices where rapid diagnosis is needed. Quantization stores numbers on a coarse integer grid, and it is the standard way to shrink such models. Its main enemy is outliers: a few very large activation values force the grid to stretch, so the ordinary values lose resolution. The usual fixes rescale or fold away the parts of the normalization layers that create those outliers. Those fixes were built for text, where outliers are artifacts tied to a few fixed feature channels. Lung sounds differ. A crackle is a brief, explosive burst, and in a spectrogram patch it may itself produce a large activation. A quantizer that flattens outliers could compress the model while silently deleting the evidence of disease.

This note proposes to first measure, then exploit, the difference between two kinds of outliers inside an Audio Spectrogram Transformer applied to lung sounds. The first kind is channel-persistent: the same few feature dimensions are large on every patch, and the normalization layers' learned scale amplifies them. The second kind is token-transient: a few time-frequency patches are large across many dimensions, possibly where crackles occur. The core claim is that these need opposite treatments. Channel-persistent outliers should be folded out of the normalization layer into the next layer's weights. Token-transient outliers should be preserved with a separate scale for each patch, and they should never be clipped. The proposed quantizer combines the two, and it is judged on how well it keeps sensitivity for the rare abnormal classes, not on overall accuracy.

Background and Gap

Two tools for normalization-stabilized quantization come from language models. Outlier Suppression showed that the learnable LayerNorm scale $\gamma$ amplifies outlier channels. It moved $\gamma$ into the following linear layer, an exact rewrite of the model, and paired this with clipping of extreme tokens. With these changes, 6-bit post-training quantization of BERT matched full precision. SmoothQuant moved per-channel difficulty from activations into weights with a fixed migration strength of 0.5, which enabled 8-bit weights and 8-bit activations (W8A8) on very large language models. Both papers motivate the proposal and supply its machinery. Both also assume that outliers are nuisance structure: Outlier Suppression clips tokens, and SmoothQuant assumes outliers persist across tokens. The token-clipping step is exactly what this proposal does not do.

Two further results constrain the design. Understanding Int4 Quantization found that Pre-LayerNorm transformers, where normalization comes before each sub-block, have wider activation ranges and are harder to quantize. The Audio Spectrogram Transformer inherits the Pre-LN design from vision transformers. The same paper and SmoothQuant note that integer matrix-multiply kernels support scales per tensor or per token, not per activation channel. Per-token scaling is therefore deployable, and per-channel activation scaling is not. Outlier-Safe Pre-Training offers a training-time alternative: a single shared RMSNorm scale and the Muon optimizer cut excess kurtosis from over 1800 to 0.04. It requires pre-training from scratch, which is unrealistic for medical models that inherit ImageNet and AudioSet weights.

On the diagnostic side, AST turns audio into spectrogram patches and transfers vision-transformer weights. VoxMed and LungAdapter show it is the strongest backbone on the ICBHI lung-sound benchmark. LungAdapter freezes the 87.53M-parameter backbone and trains 2.48M adapter parameters. Both run entirely in floating point. The only quantized lung-sound work in the file, the wearable CNN-RNN study, quantizes weights only, with a logarithmic grid, on convolutional and recurrent models, and leaves activations in floating point. The obvious next step would be "run SmoothQuant on AST." That has not been done according to the file, but it would be an incremental port. What distinguishes this proposal is the diagnostic question underneath: whether transformer outliers on lung sounds carry class information, and a quantizer designed around the answer.

Core Claim

In an AST fine-tuned on ICBHI, activation outliers split into a channel-persistent part driven by LayerNorm $\gamma$ and a token-transient part that is enriched in abnormal (crackle and "both") cycles. Standard outlier handling either clips the transient part (token clipping, percentile calibration) or spreads it across all patches through a single scale (per-tensor smoothing). Either way, it should cost much more recall on the crackle class than on normal cycles, even when the headline ICBHI score barely moves. Folding $\gamma$ into the next layer, combined with a per-patch dynamic scale and no clipping, should keep crackle recall within a small margin of full precision at W8A8, and degrade more gracefully at 4-bit weights with 8-bit activations (W4A8).

To make "split" measurable, define for layer $l$ the channel-persistence ratio

$$\rho_l = \frac{\sum_{c \in \mathcal{C}_l} \sum_t |x_{t,c}|^2}{\sum_{(t,c) \in \mathcal{O}_l} |x_{t,c}|^2},$$

where $x_{t,c}$ is the activation of patch $t$ in channel $c$, $\mathcal{O}_l$ is the set of entries in the top 0.1% by magnitude, and $\mathcal{C}_l$ is the set of channels that contain at least half of those entries. When $\rho_l$ is near 1, outliers are text-like, and folding plus smoothing should suffice. Lower values mean the energy sits in a few patches. The hypothesis is that $\rho_l$ is lower on abnormal cycles than on normal ones.

Research Program

The work has three stages. The first stage is diagnostic profiling. We fine-tune AST on ICBHI with the official patient-level split, in two variants: full fine-tuning and LungAdapter-style frozen backbone with adapters. For every block's LayerNorm output and attention-projection input, we record kurtosis (the statistic used by Outlier-Safe Pre-Training), $\rho_l$, and the magnitude of each $\gamma_c$. We then test whether patches containing outliers are over-represented in cycles labeled crackle or both, compared with normal cycles. ICBHI labels whole cycles, not frame-level onsets, so this association is tested at the cycle level, with a permutation test across patients. We also compute an attribution check: zeroing the top outlier patches should reduce the crackle logit more than zeroing random patches of equal count. If outliers turn out not to be class-enriched, the proposal falls back to an honest empirical port, and the profiling result itself becomes the finding.

The second stage builds the quantizer, which we call split-outlier quantization. Each LayerNorm $\gamma$ is folded into the following query/key/value or MLP input weights, following Outlier Suppression's exact rewrite. The patch embedding and the classification head stay at 8-bit or higher. Activations use dynamic per-token (per-patch) scales with no clipping. Weights use per-output-channel scales calibrated by minimizing reconstruction error on a calibration set balanced across classes. No step requires per-channel activation scales, so the result maps onto standard integer kernels. For the frozen-backbone variant, LungAdapter's multi-scale depthwise convolutions cannot be folded into linear weights. Because they are only 2.83% of parameters, they stay in 16-bit while the frozen backbone is quantized. This sidesteps the folding tension instead of solving it, and we report it as such.

The third stage is a clinically framed evaluation. Results are reported as the ICBHI score, per-class recall, and the drop in crackle and both-class recall relative to full precision. The last of these is the primary endpoint. A secondary transfer check fine-tunes AST on DiCOVA cough detection, scored by ROC-AUC with only 75 positives. It asks whether the same pattern appears when the positive class is very rare.

Baselines

The naive reference points are the full-precision AST and LungAdapter models from stage one, together with naive per-tensor min-max W8A8 post-training quantization. The full-precision models set the ceiling. For the strongest published number on this benchmark, LungAdapter reports an ICBHI 4-class score of 62.40%, against 59.55% for full AST fine-tuning. The wearable CNN-RNN study reports 66.31%, but on an 80-20 patient split rather than the official split, so it is not directly comparable. We will rerun it on our split if the code can be reproduced. No reported result was found on Lacuna for any activation-quantized AST on ICBHI, so our quantized baselines must be generated here.

The strongest existing method is SmoothQuant at W8A8, with migration strength 0.5 and also tuned over 0.3 to 0.8. A second strong method is Outlier Suppression, meaning $\gamma$ migration plus its token-wise clipping. These are the strongest competitors because they are the established normalization-aware post-training methods for Pre-LN transformers, and they differ from our method in exactly the place we claim matters. All methods use the same fine-tuned checkpoint and the same 256-cycle calibration set, and they receive the same hyperparameter search budget, selected on a validation fold. Each is run with both random and class-balanced calibration, so that calibration choice is not confounded with method.

The ablations isolate the key ingredient. One adds clipping back into our method. One replaces per-token with per-tensor activation scales. One removes the $\gamma$ fold. Beating the baselines means two things. First, at an equal ICBHI score (within one point), crackle and both-class recall loss must be at least halved relative to SmoothQuant and Outlier Suppression, consistently across five fine-tuning seeds. Second, if a baseline matches our recall, we report that our method adds nothing at that bit-width.

Why not just…?

Why not just run SmoothQuant off the shelf? This is the real threat. If audio outliers are channel-persistent like text outliers, SmoothQuant's per-channel migration handles them, and per-tensor scaling is enough at W8A8. The profiling stage decides the matter. If $\rho_l$ is near 1 and shows no class enrichment, we expect SmoothQuant to match our method, and we will say so. The advantage of the proposal should appear only where outliers are token-transient, and most clearly at W4A8 and under clipping-based calibration.

Why not just quantize weights only, as the wearable study did? Weight-only quantization achieved about 4× memory reduction with little loss in the wearable CNN-RNN study. However, it leaves every matrix multiplication in floating point. Understanding Int4 Quantization attributes its 8.5× latency gain to quantizing both weights and activations. We include W8A16 as a reference row. If it is accurate and our device measurements show activations do not dominate latency, the case for activation quantization weakens for that device.

Why not just use a small CNN instead of a transformer? Compact models avoid the transformer's cost. The file reports, however, that MobileNet needed the highest weight precision (Deep Neural Network for Respiratory Sound Classification in Wearable Devices Enabled by Patient Specific Model Tuning) among the lung-sound models tested. It also reports that the transformer-based LungAdapter leads the 4-class official-split comparison it reports. We compare against the rerun CNN-RNN at matched memory. If it equals the quantized AST in crackle recall, a practitioner should choose it.

Why not just run quantization-aware training on ICBHI? Quantization-aware training means training the model with the quantizer in the loop. Quadapter showed that this kind of training on small data drifts badly out of distribution, with PTB perplexity rising to about 100 against a 41 baseline. ICBHI has only 126 patients. We include LSQ-style quantization-aware training as a costlier baseline with the same epoch budget. If it beats our post-training method on held-out patients, the post-training approach is not justified.

Proposed Experiments and Decisive Checks

The first decisive check is the profiling result. If token-transient outliers are not enriched in abnormal cycles, as measured by the permutation test on $\rho_l$ and the outlier-patch ablation, then the mechanism behind the proposal is false. The second check is a stress test. Turning on clipping in any method at W8A8 should reduce crackle and both-class recall clearly more than normal-class recall. If clipping hurts all classes equally, the "outlier as signal" framing fails, even if our quantizer still works. The third check is the main comparison at W8A8 and W4A8 across five seeds, with bootstrap confidence intervals over patients. The fourth is the DiCOVA transfer, which tests whether the effect holds for a very rare positive class. Finally, we measure memory and latency with integer kernels on a CPU-class edge board, comparing the floating-point, W8A16, and W8A8 models, so that the efficiency claims rest on measurement rather than parameter counts.