Abstract: Hierarchical Multiscale LSTM (Chung et. al., 2016) is a state-of-the-art language model that learns interpretable structure from character-level input. Such models can provide fertile ground for (cognitive) computational linguistics studies. However, the high complexity of the architecture, training and implementations might hinder its applicability. We provide a detailed reproduction and ablation study of the architecture, shedding light on some of the potential caveats of re-purposing complex deep-learning architectures. We further show that simplifying certain aspects of the architecture can in fact improve its performance. We also investigate the linguistic units (segments) learned by various levels of the model, and argue that their quality does not correlate with the overall performance of the model on language modeling.
Introduction
In the pursuit of state-of-the-art (SOTA) results, deep learning architectures have grown increasingly complex, often operating as "black boxes." The Hierarchical Multiscale LSTM (HMLSTM), introduced by Chung et al. in 2016, stood out as a "white-box" alternative. It was specifically designed to learn explicit, hierarchical structures—such as words or phrases—directly from raw character input using binary boundary variables.
Despite its promise, the high complexity of the HMLSTM architecture makes it difficult to reproduce and adapt. This study aims to identify where the power of these complex systems truly resides. By systematically stripping away features and attempting to reproduce original results, we separate essential innovations from unnecessary architectural complexity.
Methodology and Reproduction
We conducted a rigorous reproduction and ablation study by porting the original TensorFlow implementation and testing several architectural variations. Our experiments focused on two standard character-level language modeling benchmarks: the Penn Treebank (PTB) for initial validation and Text8 (100 million characters from Wikipedia) for large-scale testing.
Our reproduction efforts included testing the following variants:
- Baseline Reproduction: Attempting to match original SOTA results.
- Ablation Factors: Toggling features such as Layer Normalization (LN), learning-rate schedules, and the "COPY" operation.
- Architectural Simplification:
- NoTopDown: Removing connections that pass information from higher layers back to lower layers.
- SimplerOut: Replacing the complex gated output mechanism with a simple affine transformation and ReLU.
- HMRNN: A stripped-down version replacing LSTM units with simple Elman RNNs to isolate the impact of the hierarchical structure itself.
Experimental Results
Our reproduction results on the PTB dataset were largely successful. The most faithful reproduction reached 1.29 bits-per-character (bpc), slightly trailing the original reported 1.25 bpc. However, as shown in the comparison table below, our simplified versions often outperformed the standard HMLSTM configuration.
Reproduction on the larger Text8 dataset proved more challenging, reaching 1.36 bpc compared to the original 1.29 bpc. This discrepancy likely stems from undocumented details regarding weight initialization and data pre-processing in the original publication.
Ablation Study and Architectural Insights
The ablation study revealed that "standard" optimization tricks, such as specific learning rate schedules and Layer Normalization, were often more critical to performance than the high-level architectural innovations. For instance, adding these features improved the baseline HMLSTM from 1.39 bpc to 1.27 bpc.
Surprisingly, the SimplerOut variant—which replaces the complex gated output layer—yielded the best performance on PTB at 1.25 bpc, matching the original SOTA. Furthermore, removing top-down connections (NoTopDown) only slightly degraded performance, suggesting that the model's hierarchy is primarily driven by bottom-up discovery.
Discussion: Interpretability vs. Performance
A key finding of this study is the disconnect between interpretability and performance. While the HMLSTM is designed to "discover" word boundaries, the quality of this segmentation (measured by boundary variables $z^1$ and $z^2$) does not correlate with character prediction accuracy. A model can achieve excellent language modeling results while producing linguistically nonsensical segmentations. This suggests that the model's hierarchical structure serves as a useful inductive bias for computation rather than a literal discovery mechanism for linguistic units.
Conclusion
This study highlights the "reproducibility crisis" in deep learning, where undocumented implementation "dark matter" can be as influential as the architecture itself. Our results show that simplifying the HMLSTM can lead to better performance and that its touted interpretability is not a prerequisite for its success. Future work should explore more robust gradient estimation methods, such as REINFORCE, to stabilize the learning of non-differentiable boundary variables. Ultimately, we advocate for simpler, more transparent architectures that facilitate easier community adoption and iteration.
Summary updated