Abstract
Generating extensions for finite sequences of real numbers while preserving their underlying statistical properties is a significant challenge in point process modeling. Traditional methods often fail to capture complex dependencies like repulsion or attraction between points. We present Learn2Extend, a framework utilizing a Sequence Extension Mixture Model (SEMM) to predict future points in a sequence. By combining Gated Recurrent Units (GRUs) with a log-normal mixture model, SEMM captures the conditional density of gaps, effectively maintaining both gap distributions and pair correlation functions. Our results across various datasets, including Riemann zeta zeroes and CUE eigenvalues, demonstrate superior performance over standard neural architectures.
Introduction: The Challenge of Sequence Extension
In domains such as neuroscience, finance, and physics, data is frequently represented as a point process—a sequence of discrete events. A fundamental problem is "sequence extension": given a limited sample, how can we generate additional points that maintain the original's statistical "DNA"?
Standard statistical approaches often struggle with non-independent points. For instance, in certain physical systems, points may "repel" each other, while in others, they "cluster." Simple sampling from a gap histogram ignores the crucial ordering and dependencies between these gaps.
The goal of Learn2Extend is to infer a generative model from a single sample that ensures the extended sequence preserves global statistical fingerprints, specifically the gap distribution and the pair correlation function.
Methodology: Sequence Extension Mixture Model (SEMM)
The proposed Sequence Extension Mixture Model (SEMM) shifts the focus from estimating a global intensity function to directly modeling the conditional density of the next gap.
Architecture
- Contextual Embedding: A Gated Recurrent Unit (GRU) processes the sequence history to generate a context vector, capturing local dependencies and long-term memory.
- Log-Normal Mixture Model: The model outputs parameters (weights, means, and variances) for a mixture of log-normal distributions. This multi-modal approach allows for the representation of complex probability densities where the next point might appear.
- Differentiable Sampling: Training utilizes the "Gumbel-softmax trick," making the discrete selection of mixture components differentiable and compatible with standard gradient-based optimization.
Experimental Setup
The model was rigorously tested on four types of point processes to evaluate its generalizability:
- Poisson Processes: The baseline for independent randomness.
- Circular Unit Ensemble (CUE) Eigenvalues: Characterized by local repulsion.
- Riemann $\zeta$ Function Zeroes: A mathematical sequence with structural similarities to CUE.
- Locally Attractive Processes: Gibbs point processes where points tend to cluster.
Key Results
SEMM was benchmarked against Fully Connected Neural Networks (FCNN) and standard GRUs, showing significant improvements in both accuracy and statistical fidelity.
Statistical Fidelity
A primary finding is that gaps in complex processes are not independent. While simple rejection sampling might replicate a gap histogram, it fails to preserve the pair correlation. SEMM succeeds by using its RNN component to correctly order the predicted gaps.
Performance Metrics
- Accuracy: SEMM consistently achieved lower Root Mean Squared Error (RMSE) across datasets. For Poisson sequences, SEMM's RMSE was 0.133, significantly outperforming the FCNN's 0.282.
- Riemann Zeroes: The model successfully extended sequences of Riemann zeroes. Predicted values aligned closely with actual data, maintaining the specific pair correlation structure described by Montgomery’s Pair Correlation Conjecture.
- Stability: Notably, the model exhibited minimal error propagation when predicting hundreds of terms into the future, suggesting it captures the true underlying generative logic rather than just local noise.
Conclusion and Future Work
Learn2Extend demonstrates that modeling the conditional density of gaps via mixture models is a powerful approach for sequence extension. While the model excels at handling random and repelling sequences, locally attractive processes remain a challenge, with slightly less precise statistical indicators. Future work will explore alternative mixture components and extend the framework to higher-dimensional point processes in 2D and 3D spaces.
Summary updated