Abstract: Character-level Recurrent Neural Networks (RNNs) are foundational to sequence modeling, yet the implementation details of their training and sampling phases are often overlooked. This study systematically evaluates four schemes that combine different loss calculation and sampling strategies across diverse datasets. Our findings reveal that while state transfer is theoretically advantageous, it can introduce significant training instability. Conversely, multi-loss training and progressive sampling offer substantial gains in efficiency and speed, with progressive sampling being up to 20x faster than traditional windowed methods.
Introduction
Recurrent Neural Networks (RNNs) are the workhorses of sequence modeling, yet the literature is often vague about the "plumbing" used to train them. When training on long sequences, researchers typically employ Truncated Backpropagation Through Time (TBPTT). However, the specific implementation—such as whether to calculate loss at every step or only at the end, and whether to reset the hidden state between batches—can fundamentally impact model performance and training stability.
Training and Sampling Methodologies
The authors isolate and compare four distinct schemes that combine different training and sampling strategies:
- Scheme 1 (Multi-loss + Windowed Sampling): Loss is calculated at every step in a sequence. To sample a new character, the model re-processes a "window" of the last $k$ characters from scratch.
- Scheme 2 (Single-loss + Windowed Sampling): Loss is only calculated for the very last character in a training sequence.
- Scheme 3 (Multi-loss + Progressive Sampling): Multi-loss training paired with "progressive" sampling, where the model maintains its hidden state and processes only one new character at a time.
- Scheme 4 (Conditional Multi-loss + Progressive Sampling): The most complex scheme, where the hidden state is carried over between subsequent training batches (state transfer), mimicking a continuous stream of data.
Key Insights and Results
The study, conducted across English, Finnish, Linux code, and Music datasets, yielded several critical insights:
- Instability of State Transfer: Despite its theoretical ability to learn longer dependencies, Scheme 4 proved the most unstable. It often led to diverging perplexity, particularly on the "noisier" Music dataset.
- Efficiency of Multi-loss Training: Multi-loss training provides a denser gradient signal, making it significantly more data-efficient than single-loss training. While single-loss iterations are computationally cheaper, they require far more data to reach convergence.
- Sampling Speed: Progressive sampling (Schemes 3 and 4) is up to 20x faster than windowed sampling. For an LSTM with 512 units, sampling time dropped from ~21.7ms to ~1.2ms per token.
- Data Efficiency: Reducing the step size $k_1$ between training sequences improves "label reuse," which is vital for performance when training data is scarce.
- Architecture Impact: Increasing the hidden state size provided a much larger performance boost than adding a second recurrent layer.
Conclusion
This systematic evaluation highlights that implementation details in RNNs are not merely "plumbing" but core determinants of performance. The authors recommend Scheme 3 as a robust default for most applications. Future research into hybrid loss functions—where the loss of earlier characters in a sequence is weighted less than later characters—may further enhance training stability and performance.
Summary updated