Abstract
Effective feature representation is key to the predictive performance of any algorithm. This paper introduces a meta-procedure, called Non-Euclidean Upgrading (NEU), which learns feature maps that are expressive enough to embed the universal approximation property (UAP) into most model classes while only outputting feature maps that preserve any model class's UAP. We show that NEU can learn any feature map with these two properties if that feature map is asymptotically deformable into the identity. We also find that the feature-representations learned by NEU are always submanifolds of the feature space. NEU's properties are derived from a new deep neural model that is universal amongst all orientation-preserving homeomorphisms on the input space. We derive qualitative and quantitative approximation guarantees for this architecture. We quantify the number of parameters required for this new architecture to memorize any set of input-output pairs while simultaneously fixing every point of the input space lying outside some compact set, and we quantify the size of this set as a function of our model's depth. Moreover, we show that no deep feed-forward network with commonly used activation function has all these properties. NEU's performance is evaluated against competing machine learning methods on various regression and dimension reduction tasks both with financial and simulated data.
Introduction
In modern machine learning, the efficacy of a model is often dictated by the quality of its feature representations. While Deep Neural Networks (DNNs) have achieved remarkable success, they frequently operate as "black boxes" that may inadvertently compromise the topological structure of input data. For instance, standard ReLU-based architectures can "fold" or "tear" the data space, potentially destroying critical relationships and making the original manifold irrecoverable.
This paper addresses a fundamental theoretical gap: how to ensure that a feature-learning step enhances a model's capabilities without undermining its inherent theoretical properties, such as the Universal Approximation Property (UAP). We propose the Non-Euclidean Upgrading (NEU) meta-algorithm, a "do no harm" procedure designed to upgrade simple models into universal approximators while maintaining strict geometric and topological guarantees.
The NEU Framework
The NEU meta-procedure operates through a two-stage process:
- Feature Map Learning: Training a specialized architecture known as a Reconfiguration Network.
- Model Training: Training the target hypothesis class (e.g., a linear regressor or a DNN) using the features generated in the first step.
Unlike standard neural layers that rely on affine transformations ($Ax + b$), Reconfiguration Networks utilize matrix exponentials of skew-symmetric matrices to generate rotation-based transformations. This mathematical foundation ensures that the resulting mapping is an orientation-preserving homeomorphism. Essentially, the network can stretch and twist the data space to linearize complex manifolds, but it is topologically constrained from tearing the space or merging distinct points.
To ensure generalization, we incorporate a Robustification step. This involves a re-weighting scheme based on Kullback-Leibler (KL) divergence, which minimizes the discrepancy between training and testing performance, even when the learned feature map is highly flexible.
Geometric and Topological Properties
A primary advantage of NEU is Topological Safety. The algorithm ensures that the learned feature representation remains a submanifold of the feature space. This is a significant departure from standard ReLU networks, which are not necessarily injective and thus may not be "UAP-preserving."
Furthermore, NEU demonstrates a unique capability for Targeted Memorization. It can memorize a specific set of input-output pairs while leaving the surrounding "unseen" data space largely unaffected. In contrast, standard DNNs often exhibit wild fluctuations or arbitrary "guesses" in regions between training points.
We also observe that Linearity is a Strength within this framework. NEU is particularly reliable when paired with simple base models like Ordinary Least Squares (OLS). Because the NEU component handles the complex task of manifold linearization, the simple model can operate effectively on the "upgraded" space without the internal conflicts often seen in purely deep architectures.
Experimental Evaluation
We evaluated the NEU meta-algorithm across financial datasets and simulated "pathological" functions:
- Financial Applications:
- Stock Tracking: NEU-OLS was tested against standard DNNs using Apple stock prices and a portfolio of competitors. NEU-OLS demonstrated superior out-of-sample predictive performance.
- Yield Curve Analysis: Using US government bond prices from 1990 to 2017, NEU-Autoencoders (NEU-AE) consistently outperformed PCA, Kernel-PCA, and standard Autoencoders in reconstructing yield curves, particularly in low-dimensional settings (1-2 driving factors).
- Simulated Data: We tested NEU against functions designed to challenge standard algorithms, including aperiodic oscillations ($m(x) = \cos(e^{2+x})$) and sharp jump discontinuities. In the case of discontinuities, NEU-OLS successfully captured sharp transitions that typically cause smoothing splines or LOESS models to fail.
Conclusion
The NEU meta-algorithm provides a robust framework for geometric learning, offering an explicit, low-dimensional alternative to traditional kernel methods. While highly effective, we noted potential "Kernel Clashing" when NEU features are paired with rigid kernel-based models, suggesting that NEU is best utilized with flexible or neutral hypothesis classes. Future research will explore extending this geometric approach to broader domains within Geometric Deep Learning, such as graph-based manifolds where topological preservation is paramount.
Summary updated