Deep spectral Q‐learning with application to mobile health

Yuhe Gao, Chengchun Shi, Rui Song

Abstract

Mobile health (mHealth) technologies provide a wealth of longitudinal data that can be leveraged to develop personalized Dynamic Treatment Regimes (DTRs). However, the high-dimensional and mixed-frequency nature of this data, combined with typically small patient cohorts, poses significant challenges for standard reinforcement learning algorithms. This paper presents Deep Spectral Q-learning, a framework that integrates Principal Component Analysis (PCA) with deep Q-learning to effectively manage these complexities. By applying spectral decomposition to high-frequency sensor data, the method reduces noise and prevents overfitting. Empirical evaluations on the OhioT1DM dataset and synthetic simulations demonstrate that this approach outperforms traditional methods in small-sample settings, providing more robust and effective treatment policies.

1. Introduction

The advent of wearable sensors and mobile health (mHealth) applications has transformed the landscape of precision medicine. These technologies allow for the continuous monitoring of patient vitals, such as blood glucose levels, heart rate, and physical activity. The goal is to use this data to construct Dynamic Treatment Regimes (DTRs)—sequences of decision rules that adapt treatments over time based on a patient's evolving state.

Despite the potential, mHealth data presents unique statistical challenges. Specifically, researchers often face the "Small N, Large T" problem: studies involve a limited number of participants ($N$) but generate extensive longitudinal observations ($T$). Furthermore, the data is often "mixed-frequency," where certain variables (e.g., continuous glucose monitoring) are recorded every few minutes, while others (e.g., meal intake) are recorded sporadically. Standard deep reinforcement learning models often struggle in these environments, leading to noisy decision rules and overfitting due to the high dimensionality of the state space relative to the number of independent subjects.

2. Methodology: Spectral Fitted Q-Iteration

To address these challenges, we propose the Spectral Fitted Q-Iteration algorithm. This framework combines linear dimension reduction with non-linear function approximation to derive optimal treatment policies.

2.1 Spectral Dimension Reduction

The "Spectral" component of the algorithm utilizes Principal Component Analysis (PCA) to handle high-frequency data. Rather than using raw high-dimensional inputs or simplistic summaries (like mean values), the algorithm concatenates high-frequency variables and applies spectral decomposition. By extracting the top $\kappa$ principal components, the model captures the most significant variance—representing the underlying physiological signals—while discarding high-frequency noise that could lead to overfitting.

2.2 Deep Q-Learning with Sparse Networks

The extracted principal components, alongside low-frequency covariates, serve as the input state for a deep Q-network. We employ a sparse ReLU neural network to estimate the Q-function, which represents the expected cumulative reward for taking a specific action in a given state. The use of sparsity in the network architecture further aids in regularization, ensuring the model remains generalizable despite the small number of patients.

2.3 Theoretical Foundations

A key contribution of this work is the establishment of formal regret bounds. We prove that the performance of the estimated policy converges to the true optimal policy as the sample size increases. Our analysis specifically highlights the bias-variance trade-off associated with the parameter $\kappa$: a small $\kappa$ may lead to information loss (bias), while an excessively large $\kappa$ increases the risk of fitting noise (variance).

3. Experimental Evaluation

The proposed method was evaluated using both synthetic simulations and real-world clinical data.

3.1 Synthetic Simulations

Simulations were conducted to test the algorithm's robustness across varying patient cohort sizes ($N=6$ to $N=10$) and high-frequency dimensions (up to $m=216$). The results consistently showed that Spectral Q-learning maintains performance stability where standard Deep Q-Networks (DQN) fail due to the high dimensionality of the input space.

3.2 OhioT1DM Dataset

We applied the algorithm to the OhioT1DM dataset, which contains eight weeks of data from 12 patients with Type-1 diabetes. The dataset includes:

  • High-frequency data: Continuous Glucose Monitoring (CGM) recorded every 5 minutes.
  • Low-frequency data: Self-reported meal times, exercise, and insulin dosages.

4. Results and Discussion

The Spectral Q-learning approach demonstrated superior performance in the OhioT1DM application, achieving significantly higher rewards—translated to better blood sugar management—compared to several baselines:

  1. Raw Data Models: Models using the full high-dimensional input.
  2. Simple Averaging: Models using basic statistical summaries of high-frequency data.
  3. Bottleneck Layers: Neural networks using auto-encoder style dimension reduction.

Interestingly, PCA outperformed neural bottleneck layers in small-sample settings. This suggests that for small $N$, non-parametric dimension reduction is more stable than adding more trainable parameters to a neural network. However, as $N$ increases, the performance gap narrows, as standard DQNs eventually gain enough data to process raw inputs effectively.

5. Conclusion and Future Work

Deep Spectral Q-learning provides a data-efficient solution for optimizing treatments in mHealth contexts. By balancing information retention with noise reduction through PCA, the framework enables the derivation of effective DTRs from small clinical pilot studies.

Future research will focus on relaxing the stationarity assumption of high-frequency data and exploring non-linear dimension reduction techniques, such as Variational Autoencoders (VAEs), for more complex data structures. Additionally, extending the framework to support continuous action spaces would allow for more precise clinical interventions, such as determining exact medication dosages.

Summary updated

Abstract

Dynamic treatment regimes assign personalized treatments to patients sequentially over time based on their baseline information and time‐varying covariates. In mobile health applications, these covariates are typically collected at different frequencies over a long time horizon. In this paper, we propose a deep spectral Q‐learning algorithm, which integrates principal component analysis (PCA) with deep Q‐learning to handle the mixed frequency data. In theory, we prove that the mean return under the estimated optimal policy converges to that under the optimal one and establish its rate of convergence. The usefulness of our proposal is further illustrated via simulations and an application to a diabetes dataset.
BibTeX
@inproceedings{gao2023deep,
  author = {Yuhe Gao and Chengchun Shi and Rui Song},
  title = {{Deep spectral Q‐learning with application to mobile health}},
  booktitle = {Stat},
  year = {2023},
  doi = {10.1002/sta4.564},
}
Figure