Abstract
Standard data preprocessing techniques like z-score normalization often fail to handle the complexities of real-world time series data, such as heavy tails, skewness, and multi-modality. This paper introduces Extended Deep Adaptive Input Normalization (EDAIN), a novel neural network layer that replaces static preprocessing with an end-to-end learnable normalization pipeline. EDAIN integrates adaptive outlier mitigation, shifting, scaling, and power transformations directly into the model architecture. Experiments across synthetic and real-world financial datasets demonstrate that EDAIN significantly outperforms traditional scaling methods and existing adaptive techniques by automatically optimizing the data distribution for the downstream task.
Introduction
In machine learning, data preprocessing is typically treated as a fixed, mandatory step prior to model training. However, static methods like z-score normalization or min-max scaling assume that data is relatively well-behaved. Real-world time series data frequently exhibits irregularities—heavy tails (outliers), lopsided distributions (skewness), and multiple modes—that standard scaling cannot rectify, as it only shifts and stretches the data without altering the underlying distribution's shape.
While adaptive layers like Deep Adaptive Input Normalization (DAIN) have been proposed, they primarily focus on linear shifting and scaling. EDAIN addresses this gap by incorporating non-linear transformations to handle outliers and skewness in a task-optimized manner.
The EDAIN Architecture
The EDAIN layer consists of four sequential, differentiable sublayers designed to transform "messy" input into a normalized format suitable for deep neural networks.
- Adaptive Outlier Mitigation: This sublayer uses a smoothed "winsorization" technique. By applying a
tanhfunction, it dampens the impact of extreme values while remaining differentiable, allowing the model to learn the optimal clipping threshold. - Adaptive Shifting & Scaling: These sublayers perform normalization similar to standard methods but use parameters learned via backpropagation.
- Adaptive Power Transform: This sublayer implements a learnable Yeo-Johnson transformation. This allows the model to "un-skew" data dynamically, regardless of whether the values are positive or negative.
The architecture supports two modes: Global-aware, which learns a single transformation for the entire dataset to preserve relative ordering, and Local-aware, which learns transformations specific to individual time series for handling multi-modal data.
Experimental Evaluation
The authors evaluated EDAIN across three distinct environments:
- Synthetic Datasets: Designed to test the layer's ability to handle specific irregularities like skew and multi-modality.
- Amex Default Prediction Dataset: A large-scale real-world dataset where preserving relative ordering (e.g., credit rank) is crucial.
- FI-2010 LOB Dataset: A benchmark financial dataset for Limit Order Book data, characterized by high multi-modality and non-stationarity.
Key Insights and Results
A critical finding of the study is the importance of order preservation. In the Amex dataset, "local-aware" methods—which normalize each sample independently—performed worse than simple z-scores because they destroyed the relative ranking between customers. In contrast, the "global-aware" EDAIN maintained these rankings while optimizing the distribution shape, leading to superior performance.
Performance Highlights:
- Financial Forecasting (LOB): Local-aware EDAIN achieved a Cohen’s $\kappa$ of 0.3836, significantly outperforming the previous state-of-the-art DAIN (0.3588) and BIN (0.3670).
- Credit Default (Amex): Global-aware EDAIN outperformed all other methods, proving that learning the "shape" of the normalization pays off even on unimodal data.
- Synthetic Data: EDAIN-Global achieved a Binary Cross-Entropy (BCE) loss of 0.1636, nearly matching the "Gold Standard" (CDF inversion) which requires perfect knowledge of the data-generating distribution.
Conclusion
EDAIN demonstrates that integrating preprocessing into the neural network's optimization loop provides significant benefits. By making outlier mitigation and power transforms learnable, the model can "decide" how to best transform its input to minimize final task loss. Future work includes developing hybrid layers that automatically select between global and local awareness and extending the architecture to handle missing data imputation.
Summary updated