TL;DR
This paper investigates whether $p$-adic arithmetic can serve as a "drop-in" replacement for standard floating-point math in neural network modules to mitigate numerical instability. Through experiments with linear regression, the author discovers a significant "negative result": current methods for converting between real and $p$-adic numbers are too unstable and imprecise to maintain model accuracy, failing to reconstruct even a single test prediction correctly.
What Problem Does This Paper Address?
In deep learning, particularly with recurrent neural networks, researchers often face the exploding gradient problem. When you multiply many Jacobian matrices together during backpropagation, the gradients can grow exponentially, leading to unstable training or numerical overflow.
The author explores a novel mathematical detour: $p$-adic numbers ($\mathbb{Q}_p$). Unlike the real numbers we use every day, $p$-adic numbers use a different definition of "size" (the $p$-adic norm). Crucially, they satisfy the strong triangle inequality, which implies that adding two numbers cannot produce a result larger than the largest input. Theoretically, if a neural network's operations were performed in a $p$-adic field, gradients might be naturally bounded, preventing them from "exploding."
The specific question addressed is: Can we insert a $p$-adic computational module into an existing machine-learning pipeline (Rational $\to$ $p$-adic $\to$ Computation $\to$ Rational) without breaking the model?
Methods
The author tests this "integration scheme" using a simple one-feature linear regression. The process involves:
- Forward Conversion: Mapping rational or floating-point inputs and fitted coefficients into the field of $p$-adic numbers ($\mathbb{Q}_p$).
- $p$-adic Computation: Performing the linear regression prediction ($y = ax + b$) using $p$-adic arithmetic.
- Rational Reconstruction: Using algorithms (like the extended Euclidean algorithm) to convert the $p$-adic result back into a standard rational number that the rest of the ML pipeline can understand.
The study compares two primary software implementations: the Python library pyAdic and the computer algebra system SageMath. It also tests an alternative representation based on digit reversal to see if aligning $p$-adic and real metrics more closely would help.
Datasets
The study uses the standard Diabetes dataset from scikit-learn.
- Feature: BMI (normalized, feature index 2).
- Target: Disease progression.
- Split: The last 20 observations were used as the test set to evaluate the reconstruction accuracy.
Key Insights and Ideas
- The "Strong Triangle Inequality" Hook: The primary motivation is that in $\mathbb{Q}_p$, the norm $\|u + v\|_p \leq \max(\|u\|_p, \|v\|_p)$. This suggests a world where "addition is safe," providing a potential mathematical shield against numerical instability.
- The Precision Trap: $p$-adic representations in computers are finite approximations. The author found that rational reconstruction is hyper-sensitive to the chosen precision. Small changes in precision didn't just cause small errors; they could flip the sign or change the magnitude of the result by orders of magnitude.
- The Carry Propagation Conflict: The "digit reversal" experiment provided a deep insight into why $p$-adic and real arithmetic are hard to mix. In standard addition, "carries" move toward higher powers. In a reversed $p$-adic representation, the carries would need to move in the opposite direction, meaning standard hardware addition cannot be reused.
Key Results
The results were overwhelmingly negative for the "drop-in" module approach:
- Zero Success Rate: In the main experiment ($p=2$, precision 50), 0 out of 20 test predictions were reconstructed correctly within a tolerance of $10^{-6}$.
- Reconstruction Failures: In
pyAdic, the reconstruction function frequently returnedFalse, meaning it couldn't find any rational number that matched the $p$-adic result. - Baseline Mismatch: Even when
SageMathsuccessfully completed the "round trip" conversion, the resulting values differed significantly from the baseline scikit-learn predictions. - Numerical Instability: A simple test converting the value
0.04716281to $p$-adic and back resulted in a value with the opposite sign and a different order of magnitude.
Limitations and Future Work
The author concludes that $p$-adic numbers cannot be used as a simple intermediate module in a standard real-valued pipeline. The "interface" between the two number systems is currently too lossy and unstable.
Future Directions: * Native $p$-adic Models: Instead of converting back and forth, researchers should look at models defined entirely over $p$-adic fields from the start (data, weights, and loss functions). * Automated Parameter Selection: Investigating how to automatically choose the prime $p$ and the required precision for specific tasks. * $p$-adic Cellular Neural Networks: Exploring architectures where $p$-adic digits correspond to individual neuron states, which may be better suited for parallel computation and image processing.
Summary updated