In the standard deep learning toolkit, normalization is a non-negotiable step. Whether it is Z-normalization for time series, Batch Normalization for images, or LayerNorm for Transformers, the goal is the same: squash the data into a predictable range to stabilize gradients and speed up convergence. By making models "scale-invariant," we allow them to focus on the shape of a pattern rather than its size.
However, a growing body of research suggests that this invariance comes at a steep price. In many real-world domains—from clinical diagnostics to power grid monitoring—the absolute magnitude of a signal is not noise to be filtered, but the most critical feature for understanding the underlying physics. When we normalize away the scale, we often strip the model of its ability to reason about the physical world.
The Information Loss of Invariance
The primary tension in modern architecture design is between training stability and semantic grounding. Standard normalization techniques, such as Z-normalization, effectively erase the original mean and standard deviation of a signal. As Mononito Goswami and colleagues point out in their work on MOMENT, this makes it impossible for a model to differentiate between two signals that share a shape but represent vastly different physical realities, such as a small ripple in a quiet room versus a massive surge in a power plant.
This loss of "semantic grounding" is particularly problematic for Large Language Models (LLMs) applied to numerical data. Winnie Chow and Haraldur T Hallgrimsson demonstrate that when raw magnitudes are scaled into abstract ranges, models lose the ability to apply latent world knowledge. An LLM might know that a human body temperature of 104°F is a medical emergency, but if that value is normalized to a generic "1.5," the model can no longer trigger that domain-specific reasoning (Towards Time Series Reasoning with LLMs).
When Magnitude is the Message
In specific applications, the absolute value is the primary discriminative feature. Consider these three domains where scale is essential:
- Energy Monitoring: In Non-Intrusive Load Monitoring (NILM), the goal is to identify appliances from a single power meter. Adrien Petralia and the team at EDF found that while a toaster and a space heater might have similar activation "shapes," their absolute power consumption is what distinguishes them. Their model, NILMFormer, specifically incorporates a stationarization scheme to preserve this scale.
- Clinical Biometrics: In gait analysis, the raw intensity of a sensor signal is a proxy for physical effort. Maxwell A Xu argues in RelCon that standard normalization strips away the information needed to estimate metrics like stride velocity. Similarly, in ECG analysis, the absolute amplitude of electrical activity is a vital indicator of heart health that Jintai Chen notes is often discarded by standard preprocessing (Electrocardio Panorama).
- Physical Sensing: For robotics and depth sensing, scale is synonymous with distance. Haotian Wang shows that Batch Normalization can erase the metric information provided by active sensors like LiDAR, making it difficult for models to maintain consistent depth accuracy across different environments (G2-MonoDepth).
Magnitude as a Semantic Signal
Beyond physical units, the "norm" or magnitude of a vector in latent space often carries its own semantic weight. In contrastive learning models like CLIP, researchers usually project embeddings onto a unit hypersphere, discarding the vector's length. However, Meir Yossef Levi and Guy Gilboa discovered that the raw, unnormalized magnitudes in CLIP actually encode concept frequency and uncertainty (The Double-Ellipsoid Geometry of CLIP). Common, ambiguous concepts cluster near the modality mean with smaller magnitudes, while rare, specific concepts are pushed further out.
Similarly, in Hyperbolic Contrastive Learning, Songwei Ge demonstrates that the vector norm can naturally encode image complexity or hierarchical level, a signal that is completely lost under standard spherical normalization.
Strategies for Scale-Aware Learning
To reclaim this lost information, researchers are moving toward "scale-aware" architectures that decouple training stability from information loss:
- Reversible Normalization: Used in models like RelCon, this technique normalizes data for the internal layers but retains the original scale parameters to "denormalize" the output, ensuring that regression tasks (like predicting velocity) have access to the original intensity.
- Dynamic Rescaling: Wenrui Cai and colleagues introduced SeeDNorm, which replaces static scaling with a data-dependent mechanism. By preserving the input norm information that LayerNorm typically discards, they achieve faster convergence and better zero-shot performance.
- Value-Shape Decomposition: Architectures like VSFormer by Wenjie Xi explicitly split the learning process into two paths: one that analyzes the geometric "shape" of the signal and another that preserves the raw numerical "value."
By treating magnitude as a first-class citizen rather than a nuisance variable, these approaches allow neural networks to move beyond simple pattern matching and toward a more precise, physically-grounded understanding of the data they process.