Recursive input and state estimation: A general framework for learning from time series with missing data

Alberto García-Durán, Robert West

Introduction

Real-world time-series data, such as clinical records or environmental sensor logs, are frequently plagued by missing values due to sensor failures, irregular sampling, or data filtering. Traditionally, researchers addressed this through imputation as a preprocessing step. More recently, Recurrent Neural Network (RNN) variants like GRU-D and RITS-I have been developed to handle missingness "on the fly" by adjusting internal states based on temporal gaps. However, these models often lack a common mathematical foundation. This paper introduces RISE, a unifying framework that categorizes these fragmented approaches and proposes a novel "digit-level encoding" method to improve how models represent continuous numerical values.

The RISE Framework

The authors define RISE as a general recursive architecture designed to handle missing inputs systematically. At each time step $i$, the framework employs three primary functions:

  1. Replacement ($\tilde{x}$): Determines how to estimate a missing input.
  2. Transformation ($g_x$): Maps the input (observed or replaced) into a suitable representation.
  3. Update ($g_h$): Updates the hidden state, often incorporating "discount factors" to account for the time elapsed since the last observation.
The RISE architecture unifies various RNN-based imputation models by defining explicit replacement, transformation, and update functions.

By varying these functions, the authors demonstrate that several state-of-the-art models (e.g., GRU-D, BRITS) are specific instances of the RISE framework. This unification highlights that the primary driver of success in these models is the mechanism used to decay the hidden state over time gaps.

Representation Learning: Digit-Level Encoding

A significant contribution of this work is the exploration of numerical encoders ($f_{enc}$). Instead of feeding raw scalar values into the RNN, the authors propose a Digit-level Encoder ($f_{gru}$). This approach treats numbers similarly to text:

  • A number (e.g., 123.4) is decomposed into a sequence of tokens (digits 0-9 and the decimal point).
  • These tokens are embedded and processed by a secondary "mini-RNN" (a small GRU).
  • The final hidden state of this mini-RNN serves as the high-dimensional latent representation of the number.

This method allows the model to capture non-linear relationships and "atomic" features, such as the significant digit transition between 9.9 and 10.1, which standard scalar inputs might obscure.

Experimental Evaluation

The framework was evaluated on three datasets: BG@1 and BG@5 (blood glucose levels with 39% and 82% missingness, respectively) and Air Pollution (PM10).

Key Results

The results demonstrate that the digit-level encoder ($f_{gru}$) consistently outperforms standard identity encoders ($f_{id}$) and other alternatives like binning or Transformer-style sinusoidal encoders across all RISE instances.

Performance comparison of different encoders across three datasets. The digit-level encoder ($f_{gru}$) consistently achieves the lowest error rates (MdAPE and MAPE).

In the Blood Glucose (BG@1) task, $f_{gru}$ reduced the Median Absolute Percentage Error (MdAPE) from 2.3 to 2.0. Notably, in the highly sparse BG@5 dataset, the digit-level encoder proved particularly robust, maintaining lower error rates even as the time gap since the last observation increased.

Conclusion

The RISE framework provides a much-needed theoretical unification for time-series models handling missing data. By treating numbers as sequences of digits, the authors show that models can learn more robust representations, especially in sparse data environments. While currently focused on univariate series, future work will explore multivariate applications and the computational trade-offs of this "text-like" treatment of numerical data.

Summary updated

BibTeX
@inproceedings{garciaduran2021recursive,
  author = {Alberto García-Durán and Robert West},
  title = {{Recursive input and state estimation: A general framework for learning from time series with missing data}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  doi = {10.1109/icassp39728.2021.9414801},
}
Figure