Stable Filtering for Efficient Dimensionality Reduction of Streaming Manifold Data

Nicholas P. Bertrand, Eva Yezerets, Han Lun Yap, Adam Charles, Christopher J. RozellarXiv 2026 · DOI · arXiv

TL;DR

This paper introduces Randomized Filtering (RF), a computationally efficient, data-independent method for compressing high-dimensional streaming data. By leveraging randomized linear algebra, RF provably preserves the underlying non-linear geometry (the "manifold") of scientific data, allowing researchers to perform complex analyses like event detection and manifold learning directly on compressed streams.

What Problem Does This Paper Address?

We are currently in a "data deluge." Modern sensors—from high-resolution brain probes to fluid dynamics cameras—generate data at rates that far exceed our ability to store, transmit, or process it in real-time.

Traditional dimensionality reduction tools like Principal Component Analysis (PCA) or Variational Autoencoders (VAEs) have significant drawbacks for streaming scientific data:

  1. Data Dependency: They require a "training" phase to learn the structure of the data before they can compress it. In discovery-based science, we often don't have this luxury.
  2. Computational Cost: Training these models is expensive and slow, making them unsuitable for low-power, on-board hardware.
  3. Geometry Loss: PCA focuses on maximizing variance, which might inadvertently discard the subtle, non-linear geometric structures (manifolds) that contain the actual scientific insights.

The authors seek a "universal" compressor that is fast, requires no training, and maintains the geometric integrity of the data.

Methods

The authors propose Randomized Filtering (RF), which is a practical implementation of the Fast Johnson-Lindenstrauss (JL) transform tailored for manifold data. The algorithm consists of three simple, hardware-friendly steps:

  1. Randomize Signs: Multiply the input vector by a random sequence of $\pm 1$ (a Rademacher matrix).
  2. Compute FFT: Apply the Fast Fourier Transform (FFT).
  3. Randomly Subsample: Pick a random subset of the resulting Fourier coefficients to keep.

The core "magic" here is the Restricted Isometry Property (RIP). The authors provide theoretical proofs (Theorem 1 and 2) showing that this specific randomized mapping acts as a "stable embedding." This means that the distances between points on the original high-dimensional manifold are approximately preserved in the new, lower-dimensional space. Because the FFT is the heaviest lift, the algorithm runs in $O(N \log N)$ time, making it incredibly fast.

Datasets

The paper evaluates RF across a diverse range of synthetic and real-world datasets:

  • Synthetic:
    • A 1D "Sine Manifold" (complex exponential curve) used for theoretical validation.
  • A cardiac electrophysiology model (OVVR) simulating heart failure.
  • Simulated calcium imaging data with overlapping neurons and noise.
  • Solutions to 2D vorticity equations (fluid dynamics).
  • Real-World:
    • Mouse Neural Fluorescence: A massive dataset of 6,519 neurons recorded at 9.6 Hz (Manley et al.).
  • Voltage-Sensitive Dye (VSD) Imaging: Recordings from rodent primary sensory cortex.
  • fMRI: The Enhanced Nathan Kline Institute-Rockland Sample dataset.

Key Insights and Ideas

  • Data-Independence is a Feature, Not a Bug: Because RF doesn't "learn" from the data, it is robust to "distribution shift." If the underlying system changes (e.g., a brain state shifts or fluid flow becomes turbulent), RF continues to work without needing to be retrained.
  • Preserving Geometry > Preserving Variance: The authors show that even when PCA captures more "variance," RF often does a better job of preserving the shape of the data manifold. This is critical for downstream tasks like Locally Linear Embedding (LLE).
  • The "Discovery" Advantage: In many experiments, we don't know which features are important a priori. RF’s universal nature ensures that we don't accidentally filter out the very phenomena we are trying to discover.
  • Hardware Potential: The authors point out that the FFT can be implemented optically (using a lens), suggesting a future where compression happens at the speed of light before the data even hits a digital sensor.

Key Results

  • Stability: RF achieved significantly lower "isometry constants" (a measure of distortion) compared to standard low-pass filtering (LPF) across all datasets.
  • Neural Activity Recovery: In simulated calcium imaging, RF maintained an F1-score above 0.9 for event detection even at 40x–80x compression. LPF performance dropped sharply at much lower compression ratios.
  • Manifold Preservation: On real mouse brain data, RF preserved the 4D manifold shape (measured via Procrustes distance) better than PCA for compression ratios up to 100x.
  • Fluid Dynamics: RF allowed for accurate classification of vortex phases even under heavy noise, whereas LPF-compressed data became unclassifiable.

Limitations and Future Work

  • Parameter Tuning: While RF is data-independent, the user still needs to choose the target dimension ($M$). While the theory provides bounds, in practice, this still requires some trial and error or knowledge of the manifold's intrinsic dimension.
  • Reconstruction: This paper focuses on analysis in the reduced space. While the geometry is preserved, perfectly reconstructing the original high-dimensional signal from the RF-compressed version (e.g., for visualization) is a separate, more difficult inverse problem.
  • Scaling: While tested on thousands of neurons, the authors look toward "million-neuron" datasets to see if the logarithmic scaling predicted by their theory holds up in extreme regimes.

Summary updated

Abstract

Many areas in science and engineering now have access to technologies that enable the rapid collection of overwhelming data volumes. While these datasets are vital for understanding phenomena from physical to biological and social systems, the sheer magnitude of the data makes even simple storage, transmission, and basic processing highly challenging. To enable efficient and accurate execution of these data processing tasks, we require new dimensionality reduction tools that 1) do not need expensive, time-consuming training, and 2) preserve the underlying geometry of the data that has the information required to understand the measured system. Specifically, the geometry to be preserved is that induced by the fact that in many applications, streaming high-dimensional data evolves on a low-dimensional attractor manifold. Importantly, we may not know the exact structure of this manifold a priori. To solve these challenges, we present randomized filtering (RF), which leverages a specific instantiation of randomized dimensionality reduction to provably preserve non-linear manifold structure in the embedded space while remaining data-independent and computationally efficient. In this work we build on the rich theoretical promise of randomized dimensionality reduction to develop RF as a real, practical approach. We introduce novel methods, analysis, and experimental verification to illuminate the practicality of RF in diverse scientific applications, including several simulated and real-data examples that showcase the tangible benefits of RF.
BibTeX
@misc{bertrand2026stable,
  author = {Nicholas P. Bertrand and Eva Yezerets and Han Lun Yap and Adam Charles and Christopher J. Rozell},
  title = {{Stable Filtering for Efficient Dimensionality Reduction of Streaming Manifold Data}},
  year = {2026},
  eprint = {2601.08685},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.08685},
}
Figure