In deep learning, data normalization is an almost universal reflex. Whether utilizing Z-score normalization or instance normalization, forcing inputs into a zero-mean, unit-variance distribution stabilizes gradients, accelerates convergence, and mitigates distribution shifts.
However, this mathematical convenience comes with a hidden physical cost: it strips away the absolute scale of the signal.
When we normalize a time-series signal, we preserve its relative shape but erase its absolute magnitude. For abstract data like text or images, this is often desirable—a geometric shape remains recognizable regardless of its size or lighting brightness. But for physical, physiological, and engineering signals, magnitude is not noise; it is the core semantic content. Forcing a whisper and a shout into the same variance range makes them indistinguishable to a neural network.
A growing body of research addresses this "normalization-induced information loss," developing techniques to maintain training stability without blinding models to the real-world scale of physical data.
The Cost of Scale Blindness
To understand why discarding scale is problematic, consider how machine learning models interact with physical environments. In these domains, absolute thresholds dictate physical reality.
In clinical diagnostics, the raw voltage of an electrical signal or the absolute velocity of fluid dynamics carries vital diagnostic markers:
- Electrocardiograms (ECGs): Aruna Mohan et al. demonstrate in Deciphering Heartbeat Signatures: A Vision Transformer Approach to Explainable Atrial Fibrillation Detection from ECG Signals that retaining raw signal amplitude preserves critical voltage variations essential for detecting conditions like atrial fibrillation. When standard z-normalization is applied, these absolute voltage levels are lost, forcing a trade-off between training stability and feature preservation. Similarly, Jintai Chen et al. note in Electrocardio Panorama: Synthesizing New ECG views with Self-supervision that standard signal normalization inadvertently discards absolute amplitude information that is critical for accurate clinical interpretation.
- Blood Flow Dynamics: In transcranial Doppler ultrasound analysis, Samuel Thorpe et al. found in Toward Automated Classification of Pathological Transcranial Doppler Waveform Morphology via Spectral Clustering that normalizing signal data to a standard scale removes absolute magnitude features like flow velocity. This prevents clustering algorithms from identifying clinical categories that rely on absolute physical thresholds rather than relative wave shapes.
- Biometrics and Motion: In health monitoring via wearables, Maxwell A Xu et al. point out in RelCon: Relative Contrastive Learning for a Motion Foundation Model for Wearable Data that standard pipelines strip away scale, yet the raw magnitude of a motion signal is the primary indicator for quantitative metrics like gait velocity or physical exertion.
Outside of medicine, the problem is equally pronounced. In radar signal classification, Seok‐Jun Hong et al. show in Automatic Classification of Radar Signals Using CNN that normalization removes absolute frequency ranges, making it difficult to differentiate signals that share a temporal pattern but operate at entirely different physical magnitudes. In wireless communications, Eunju Seo et al. argue in Normalization-Free Knowledge Distillation for Variable-Bit CSI Feedback in Massive MIMO that forcing a zero-mean and unit-variance distribution makes models blind to the energy distribution necessary for accurate signal reconstruction, discarding critical physical features like path loss and antenna gain.
Raw Signal (Whisper vs. Shout) Standard Normalization (Information Loss)
^ ^
| _ | _ _
| / \ | / \ / \
| / \ _ | / \ / \
| / \ / \ | / \ / \
+--/-------\--/---\----------> +--/-------\------/-------\---------->
[Shout] [Whisper] [Shout] [Whisper]
(High Amp) (Low Amp) (Both scaled to unit variance)
To resolve this tension, researchers have developed three primary paradigms: Reversible Normalization, Statistical Re-injection, and Normalization-Free Learning.
Paradigm 1: Reversible Normalization
In time-series forecasting, models must handle non-stationary data where the mean and variance drift over time (distribution shift). While instance normalization stabilizes training by centering each sequence, it makes it impossible for a model to predict the actual future values because the absolute scale has been discarded.
To solve this, Taesung Kim et al. proposed Reversible Instance Normalization (RevIN) in Reversible Instance Normalization for Accurate Time-Series Forecasting against Distribution Shift.
The mechanism operates like a mathematical bracket around the neural network:
$$\text{Raw Input } X \longrightarrow \boxed{\text{Normalize (Save } \mu, \sigma)} \longrightarrow \boxed{\text{Neural Network Layers}} \longrightarrow \boxed{\text{Denormalize (Apply } \mu, \sigma)} \longrightarrow \text{Scale-Accurate Forecast}$$
- Normalization: The input sequence is normalized to a zero-mean and unit-variance distribution before being fed into the model layers, stripping the distribution shift.
- Side-channel: The original mean ($\mu$) and standard deviation ($\sigma$) are saved and bypassed around the internal layers.
- Denormalization: After the model computes its normalized forecast, the saved statistics are used to apply the inverse mathematical operation.
This architecture allows the model's internal layers to operate in a stable, normalized space while ensuring the final output layer restores the absolute physical scale necessary for real-world reconstruction.
Paradigm 2: Statistical Re-injection
As Time-Series Foundation Models (TSFMs) become more prominent, researchers face a new challenge: a model pre-trained to forecast (which benefits from normalization) is often repurposed to classify signals (where absolute scale is a key discriminative feature). When the foundation model's embeddings discard scale, downstream tasks suffer.
To fix this without retraining the core model, researchers use statistical re-injection—explicitly passing the lost metadata back into the downstream components.
+---> [ Normalization ] ---> [ Frozen TSFM Encoder ] ---+
| v
Raw Time Series --+ [ Concatenate ] ---> Classifier
| ^
+---> [ Calculate Mean & Variance ] --------------------+
In Pre-trained Forecasting Models: Strong Zero-Shot Feature Extractors for Time Series Classification, Andreas Auer et al. show that while instance normalization stabilizes forecasting, it discards the absolute sample statistics needed for classification. The authors demonstrate that simply supplementing the frozen model's features with the raw sample statistics (mean and variance) restores these discriminative cues and significantly improves classification performance.
Similarly, Yujin Lee et al. explore this in Time-Series Foundation Model Embeddings as Means for Physiological Feature Extraction. They show that re-injecting global scale statistics into TSFM embeddings helps recover lost amplitude information, though they note that simple re-injection may still struggle to capture all subtle, localized amplitude variations.
This concept also bridges the gap between continuous signals and Large Language Models (LLMs). In OpenTSLM: Time-Series Language Models for Reasoning over Multivariate Medical Text- and Time-Series Data, Patrick Langer et al. note that standard encoders strip away the absolute physical scale of measurements. To maintain context for clinical reasoning, the authors manually reintroduce absolute statistical parameters like the mean and standard deviation directly into the textual prompt, allowing the LLM to understand the real-world significance of the underlying data values.
Paradigm 3: Normalization-Free Learning
Instead of normalizing data and attempting to recover the lost scale later, a final paradigm avoids standard normalization entirely. This approach requires modifying the training process or loss functions to handle raw scales directly without causing gradient explosion.
This is particularly vital in Knowledge Distillation (KD), where a smaller student network learns from a larger teacher network. Standard KD relies on Z-score normalization to align the tensor distributions of the teacher and student. However, in domains like wireless communications, this alignment erases physical reality.
To solve this, Eunju Seo et al. developed Normalization-Free Knowledge Distillation (NF-KD) in Normalization-Free Knowledge Distillation for Variable-Bit CSI Feedback in Massive MIMO.
Instead of forcing a zero-mean and unit-variance distribution, NF-KD applies divergence losses directly to raw, temperature-scaled tensors. By preserving the raw scale of tensors during training, the student model successfully inherits the absolute magnitude of the input signals, retaining the energy distribution and path loss characteristics necessary for accurate signal reconstruction.
Summary of Approaches
| Strategy | Core Mechanism | Primary Use Case | Key Benefit |
|---|---|---|---|
| Reversible Normalization | Normalizes inputs, caches statistics, and applies inverse transformation at output. | Time-series forecasting under distribution shift. | Maintains internal training stability while outputting exact physical values. |
| Statistical Re-injection | Appends raw mean/variance to frozen embeddings or text prompts. | Downstream classification using frozen foundation models. | Restores discriminative scale cues without retraining the base model. |
| Normalization-Free Learning | Eliminates normalization layers; applies losses directly to raw, scaled tensors. | Knowledge distillation and physical signal reconstruction. | Preserves absolute energy distributions and physical properties natively. |
Key Foundational Work
Toward Automated Classification of Pathological Transcranial Doppler Waveform Morphology via Spectral Clustering (2019) — contributed 1 concept elements to this direction.
Automatic Classification of Radar Signals Using CNN (2019) — contributed 1 concept elements to this direction.
Electrocardio Panorama: Synthesizing New ECG views with Self-supervision (2021) — contributed 1 concept elements to this direction.