Enabling hyperparameter optimization in sequential autoencoders for spiking neural data

Mohammad Reza Keshtkaran, Chethan PandarinathNeurIPS 2019

TL;DR

This paper introduces two novel techniques—sample validation and coordinated dropout—to solve a "hidden" overfitting problem in sequential autoencoders (SAEs) used for neural data. By preventing the model from simply memorizing spike times, these methods enable large-scale hyperparameter optimization. This allows complex models like LFADS to maintain high performance even with 10x less training data than previously required.

What Problem Does This Paper Address?

In neuroscience, researchers use Sequential Autoencoders (SAEs), such as LFADS (Latent Factor Analysis via Dynamical Systems), to uncover the "latent dynamics" hidden within the noisy spiking activity of hundreds of neurons. While powerful, these models are massive, often containing hundreds of thousands of parameters.

Sequential Autoencoder architecture and the overfitting problem.

The authors identify a critical roadblock: SAEs are prone to a specific type of overfitting where the network learns a trivial "identity transformation." Instead of uncovering the underlying biological dynamics, the model learns to pass raw spike times directly from input to output.

Crucially, standard validation metrics cannot detect this. Because the model "sees" the spikes during the inference step to produce a reconstruction, it can achieve a near-perfect reconstruction loss without actually learning anything useful about the system's dynamics. This makes it impossible to use automated hyperparameter (HP) searches, forcing researchers to rely on tedious, manual "hand-tuning."

Methods

To break the identity mapping and enable automated HP optimization, the authors propose two complementary solutions:

Illustration of Sample Validation and Coordinated Dropout methods.
  1. Sample Validation (SV): Unlike standard validation which holds out entire trials, SV uses a "speckled holdout" pattern. It randomly masks individual data points (specific neurons at specific times) within a trial. The model must then predict these missing values. Because the model never sees the held-out samples during training or inference, the reconstruction error on these samples provides an honest metric of how well the model has captured the shared structure of the data.
  2. Coordinated Dropout (CD): This is a novel regularization strategy. During training, a random mask is applied to the input. Simultaneously, the complement of that mask is used to block gradients at the output. In plain English: the network is forced to reconstruct a data point using only information from other data points. This physically prevents the network from learning an identity mapping.
  3. Population Based Training (PBT): With a reliable validation metric (SV) and a robust regularizer (CD) in hand, the authors integrated LFADS with PBT—an evolutionary algorithm that trains a "population" of models in parallel, automatically exploring and refining hyperparameters like learning rates and L2 penalties.

Datasets

The researchers evaluated their methods on both synthetic and real-world biological data:

  • Synthetic RNN Dataset: An input-driven recurrent neural network used to generate "ground truth" firing rates, allowing for precise measurement of model accuracy ($R^2$).
  • Monkey J Maze: A large dataset (2,296 trials) of neural activity from the primary motor cortex (M1) and dorsal premotor cortex (PMd) recorded while a macaque performed reaching movements.
  • Random Target Task (RTT): A publicly available dataset featuring continuous, self-paced reaches to targets in a grid.

Key Insights and Ideas

  • The Identity Trap: The most surprising insight is that a model can look "perfect" on paper (low reconstruction loss) while being scientifically useless. Spiking data is sparse enough that a high-capacity RNN can easily "cheat" by memorizing noise.
  • Forcing Collaboration: Coordinated Dropout is a clever "anti-cheating" mechanism. By dictating that the model cannot use a specific neuron's spike at a specific time to predict that exact same spike, the model is forced to find the underlying latent factors that the neuron shares with its neighbors.
  • Data Efficiency via Optimization: The perceived need for massive datasets in deep learning is often just a symptom of poor hyperparameter settings. By automating the search, the authors showed that SAEs are far more data-efficient than the field previously believed.

Key Results

  • 10x Data Reduction: Using HP optimization with Coordinated Dropout, the authors achieved performance on only 10% of the training data that was comparable to what fixed-HP models achieved on the full dataset.
  • Reliable Metrics: Sample Validation showed a clear, strong correlation with the actual ground-truth accuracy ($R^2$), whereas standard validation loss was almost completely uninformative for overfit models.
  • Improved Decoding: In the Monkey J Maze task, HP-optimized models significantly outperformed fixed-HP models in decoding the monkey's hand velocity, especially as the number of available training trials decreased.

Limitations and Future Work

The authors note that Coordinated Dropout acts as a very strong regularizer. While this prevents overfitting, it can potentially lead to underfitting if the data is extremely sparse or if the number of observed dimensions (neurons) is very small relative to the complexity of the latent space.

Future work remains to determine the optimal "keep ratio" for dropout across different brain regions and to test if these techniques can be applied to other types of sparse, multidimensional time-series data outside of neuroscience, such as financial forecasting or climate modeling.

Summary updated

BibTeX
@inproceedings{keshtkaran2019enabling,
  author = {Mohammad Reza Keshtkaran and Chethan Pandarinath},
  title = {{Enabling hyperparameter optimization in sequential autoencoders for spiking neural data}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
}
Figure