Character-level Recurrent Neural Networks in Practice: Comparing Training and Sampling Schemes

Cedric De Boom, Thomas Demeester, Bart Dhoedt

Abstract: Character-level Recurrent Neural Networks (RNNs) are foundational to sequence modeling, yet the implementation details of their training and sampling phases are often overlooked. This study systematically evaluates four schemes that combine different loss calculation and sampling strategies across diverse datasets. Our findings reveal that while state transfer is theoretically advantageous, it can introduce significant training instability. Conversely, multi-loss training and progressive sampling offer substantial gains in efficiency and speed, with progressive sampling being up to 20x faster than traditional windowed methods.

Introduction

Recurrent Neural Networks (RNNs) are the workhorses of sequence modeling, yet the literature is often vague about the "plumbing" used to train them. When training on long sequences, researchers typically employ Truncated Backpropagation Through Time (TBPTT). However, the specific implementation—such as whether to calculate loss at every step or only at the end, and whether to reset the hidden state between batches—can fundamentally impact model performance and training stability.

Unrolling a Recurrent Neural Network (RNN) to visualize its sequential processing of input $x_n$ to predict $x_{n+1}$.

Training and Sampling Methodologies

The authors isolate and compare four distinct schemes that combine different training and sampling strategies:

  1. Scheme 1 (Multi-loss + Windowed Sampling): Loss is calculated at every step in a sequence. To sample a new character, the model re-processes a "window" of the last $k$ characters from scratch.
  2. Scheme 2 (Single-loss + Windowed Sampling): Loss is only calculated for the very last character in a training sequence.
  3. Scheme 3 (Multi-loss + Progressive Sampling): Multi-loss training paired with "progressive" sampling, where the model maintains its hidden state and processes only one new character at a time.
  4. Scheme 4 (Conditional Multi-loss + Progressive Sampling): The most complex scheme, where the hidden state is carried over between subsequent training batches (state transfer), mimicking a continuous stream of data.
Truncated Backpropagation Through Time (TBPTT) parameters, showing the truncation window $k_2$ and the step size $k_1$.

Key Insights and Results

The study, conducted across English, Finnish, Linux code, and Music datasets, yielded several critical insights:

  • Instability of State Transfer: Despite its theoretical ability to learn longer dependencies, Scheme 4 proved the most unstable. It often led to diverging perplexity, particularly on the "noisier" Music dataset.
  • Efficiency of Multi-loss Training: Multi-loss training provides a denser gradient signal, making it significantly more data-efficient than single-loss training. While single-loss iterations are computationally cheaper, they require far more data to reach convergence.
  • Sampling Speed: Progressive sampling (Schemes 3 and 4) is up to 20x faster than windowed sampling. For an LSTM with 512 units, sampling time dropped from ~21.7ms to ~1.2ms per token.
  • Data Efficiency: Reducing the step size $k_1$ between training sequences improves "label reuse," which is vital for performance when training data is scarce.
  • Architecture Impact: Increasing the hidden state size provided a much larger performance boost than adding a second recurrent layer.

Conclusion

This systematic evaluation highlights that implementation details in RNNs are not merely "plumbing" but core determinants of performance. The authors recommend Scheme 3 as a robust default for most applications. Future research into hybrid loss functions—where the loss of earlier characters in a sequence is weighted less than later characters—may further enhance training stability and performance.

Summary updated

Abstract

Recurrent neural networks are nowadays successfully used in an abundance of applications, going from text, speech and image processing to recommender systems. Backpropagation through time is the algorithm that is commonly used to train these networks on specific tasks. Many deep learning frameworks have their own implementation of training and sampling procedures for recurrent neural networks, while there are in fact multiple other possibilities to choose from and other parameters to tune. In the existing literature, this is very often overlooked or ignored. In this paper, we therefore give an overview of possible training and sampling schemes for character-level recurrent neural networks to solve the task of predicting the next token in a given sequence. We test these different schemes on a variety of datasets, neural network architectures and parameter settings, and formulate a number of take-home recommendations. The choice of training and sampling scheme turns out to be subject to a number of trade-offs, such as training stability, sampling time, model performance and implementation effort, but is largely independent of the data. Perhaps the most surprising result is that transferring hidden states for correctly initializing the model on subsequences often leads to unstable training behavior depending on the dataset.
BibTeX
@article{boom2018character,
  author = {Cedric De Boom and Thomas Demeester and Bart Dhoedt},
  title = {{Character-level Recurrent Neural Networks in Practice: Comparing Training and Sampling Schemes}},
  journal = {Neural Computing and Applications},
  year = {2018},
  url = {https://openreview.net/forum?id=GDMomB1yMZ},
}
Figure