A journey in ESN and LSTM visualisations on a language task

Alexandre Variengien, Xavier HinautarXiv 2020 · arXiv

Introduction

Recurrent Neural Networks (RNNs) are powerful tools for processing sequential data, yet their internal mechanisms for representing linguistic structure often remain opaque. In "A Journey in ESN and LSTM Visualisations on a Language Task," Variengien and Hinaut investigate the internal dynamics of two distinct RNN architectures: the Long Short-Term Memory (LSTM) and the Echo State Network (ESN). By employing a psycholinguistic task and novel visualization techniques, the authors shed light on how these models organize symbolic information.

The Cross-Situational Learning Task

To compare these architectures, the researchers utilized a Cross-Situational Learning (CSL) task. This paradigm mimics early language acquisition in infants, where a learner must associate heard sentences with visual scenes containing multiple objects. The model is tasked with predicting the visual properties (color, position, category) of objects described in a sentence without explicit word-to-object mapping.

Overview of the Cross-Situational Learning (CSL) framework, illustrating how linguistic input is mapped to an imagined perceptual representation.

The study used a synthetic corpus generated by a context-free grammar. Sentences described one or two objects, and the models were evaluated on their ability to produce a vector representing the correct visual properties. This setup allowed for controlled scaling of vocabulary size to test model robustness.

Architectural Approaches: LSTM vs. ESN

The study pits the standard LSTM against the Echo State Network (ESN), a form of Reservoir Computing. While LSTMs are trained entirely via Back-propagation Through Time (BPTT), ESNs utilize a fixed, random recurrent "reservoir."

Schematic of an Echo State Network (ESN), highlighting the fixed reservoir weights and the trainable readout layer.

In the ESN framework, only the weights of the readout layer are trained, typically using the FORCE learning algorithm. This approach is often considered more biologically plausible and computationally efficient than full BPTT, as the internal recurrent weights remain fixed after initialization.

Visualizing Internal States with RSSviz

A core contribution of the paper is the introduction of Recurrent States Space Visualisations (RSSviz). This technique uses UMAP (Uniform Manifold Approximation and Projection) to project high-dimensional hidden states into a 2D space. By coloring these points based on input words or output predictions, the researchers can observe the structure of the network's memory.

Fractal Topology and Structural Pivots

RSSviz revealed that LSTMs naturally develop a "fractal" organization of their state space. Hidden states are clustered into "archipelagos" representing feature combinations, with finer sub-structures emerging upon closer inspection. This suggests a hierarchical organization of recursive information.

Both models also converged on a similar strategy for handling complex sentences: the "And" switch. The word "and" serves as a structural pivot, transitioning the model's internal state from the region representing the first object to the region for the second object.

Key Results and Performance

  • Efficiency: The ESN proved significantly faster, training nearly 10 times quicker than the LSTM (47.1s vs. 410s) while maintaining comparable performance on small datasets (0.21% vs 0.037% error).
  • Robustness to Scaling: While the LSTM achieved higher precision on simple tasks, its performance degraded sharply as vocabulary size increased. In contrast, the ESN demonstrated superior robustness, maintaining low error rates without extensive hyperparameter tuning as the number of objects scaled from 4 to 50.
  • Uncertainty Mapping: RSSviz allowed for the identification of "dim" clusters in the ESN state space, which corresponded directly to linguistic ambiguities in the input, such as confusing spatial descriptors like "middle."

Conclusion and Future Directions

While the current analysis is primarily qualitative, it demonstrates that even random reservoirs in ESNs can effectively encode complex sentence structures. Future work aims to apply RSSviz to natural language datasets to see if similar fractal structures emerge in less controlled environments. Additionally, developing quantitative metrics for these visualizations could turn RSSviz into a powerful diagnostic tool for RNN development, helping engineers understand why models fail on specific edge cases.

Summary updated

Abstract

Echo States Networks (ESN) and Long-Short Term Memory networks (LSTM) are two popular architectures of Recurrent Neural Networks (RNN) to solve machine learning task involving sequential data. However, little have been done to compare their performances and their internal mechanisms on a common task. In this work, we trained ESNs and LSTMs on a Cross-Situationnal Learning (CSL) task. This task aims at modelling how infants learn language: they create associations between words and visual stimuli in order to extract meaning from words and sentences. The results are of three kinds: performance comparison, internal dynamics analyses and visualization of latent space. (1) We found that both models were able to successfully learn the task: the LSTM reached the lowest error for the basic corpus, but the ESN was quicker to train. Furthermore, the ESN was able to outperform LSTMs on datasets more challenging without any further tuning needed. (2) We also conducted an analysis of the internal units activations of LSTMs and ESNs. Despite the deep differences between both models (trained or fixed internal weights), we were able to uncover similar inner mechanisms: both put emphasis on the units encoding aspects of the sentence structure. (3) Moreover, we present Recurrent States Space Visualisations (RSSviz), a method to visualize the structure of latent state space of RNNs, based on dimension reduction (using UMAP). This technique enables us to observe a fractal embedding of sequences in the LSTM. RSSviz is also useful for the analysis of ESNs (i) to spot difficult examples and (ii) to generate animated plots showing the evolution of activations across learning stages. Finally, we explore qualitatively how the RSSviz could provide an intuitive visualisation to understand the influence of hyperparameters on the reservoir dynamics prior to ESN training.
BibTeX
@misc{variengien2020journey,
  author = {Alexandre Variengien and Xavier Hinaut},
  title = {{A journey in ESN and LSTM visualisations on a language task}},
  year = {2020},
  eprint = {2012.01748},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.01748},
}
Figure