Abstract
Dynamic Boltzmann machines (DyBMs) are recently developed generative models of a time series. They are designed to learn a time series by efficient online learning algorithms, whilst taking long-term dependencies into account with help of eligibility traces, recursively updatable memory units storing descriptive statistics of all the past data. The current DyBMs assume a finite-dimensional time series and cannot be applied to a functional time series, in which the dimension goes to infinity (e.g., spatiotemporal data on a continuous space). In this paper, we present a functional dynamic Boltzmann machine (F-DyBM) as a generative model of a functional time series. A technical challenge is to devise an online learning algorithm with which F-DyBM, consisting of functions and integrals, can learn a functional time series using only finite observations of it. We rise to the above challenge by combining a kernel-based function approximation method along with a statistical interpolation method and finally derive closed-form update rules. We design numerical experiments to empirically confirm the effectiveness of our solutions. The experimental results demonstrate consistent error reductions as compared to baseline methods, from which we conclude the effectiveness of F-DyBM for functional time series prediction.
Introduction
Most classic time-series models, such as Vector Autoregression (VAR) or standard Dynamic Boltzmann Machines, assume a fixed, finite number of dimensions—essentially, a static set of sensors at fixed locations. However, real-world spatiotemporal data often breaks these assumptions. We frequently encounter scenarios with infinite potential locations where we might want to forecast values at arbitrary points in continuous space. Furthermore, in IoT or mobile applications, sensors may move or appear and disappear, making it impossible to construct a consistent "vector" of inputs over time. Finally, we rarely have a complete "image" of a functional pattern, only finite, scattered observations.
The Functional Dynamic Boltzmann Machine (F-DyBM) addresses these challenges by treating a time series as a sequence of functions rather than vectors. It provides a lightweight model capable of online training to handle functional time series while maintaining high predictive performance.
Methodology
The F-DyBM extends the standard DyBM framework into the functional domain. To make this computationally feasible, the authors introduce several key techniques:
- From Neurons to Fields: Discrete neural units are replaced with an infinite number of neurons distributed across a space, modeled using a Gaussian Process (GP).
- Weight Functions and Integrals: Instead of weight matrices, the model uses weight functions. The standard summation of past states becomes an integral over the spatial domain.
- The Kernel Trick: By assuming weight functions have specific parametric forms based on kernels, the authors use the kernel trick to derive closed-form, analytical update rules.
- Eligibility Traces: The model utilizes recursively updated memory units that store a running summary of all past data. This allows it to capture long-term dependencies without the heavy overhead of Backpropagation Through Time (BPTT).
- MAP Estimation for Imputation: Since only finite data points are available at each time step, the model uses Maximum-a-Posteriori (MAP) estimation to estimate the full functional pattern before updating its internal memory.
Experimental Evaluation
The F-DyBM was evaluated on five real-world spatiotemporal datasets, including NOAA Global Surface Temperature anomalies and AirData from the US EPA (CO, NO2, O3, and SO2). These datasets are particularly challenging because sensor locations and counts vary over time.
The model was compared against four baselines: Vector Autoregression (VAR), Functional Autoregression (FAR), and the standard Gaussian DyBM (G-DyBM). F-DyBM achieved the lowest Root-Mean-Square Error (RMSE) across all five datasets. The inclusion of eligibility traces for long-term memory decreased error by an average of 12%, while the functional modeling approach decreased error by an additional 11.7%.
Conclusion
F-DyBM successfully extends the DyBM framework to infinite-dimensional functional time series. By treating space as a continuum, the model is robust to sensor flux and movement. While the current implementation faces the $O(N^3)$ complexity bottleneck typical of Gaussian Processes, future work could incorporate sparse GP techniques to scale to massive networks. Additionally, the authors propose applying F-DyBM to new domains, such as treating word vectors in language modeling as positions in a continuous semantic space.
Summary updated