TreeESN: a Preliminary Experimental Analysis

Claudio Gallicchio, Alessio MicheliESANN 2010 · OpenReview

Introduction

Recursive Neural Networks (RecNNs) are powerful tools for processing structured data, such as chemical molecules or natural language parse trees. However, training them is computationally expensive because gradients must be backpropagated through the entire structure. This paper introduces the Tree Echo State Network (TreeESN), which extends the efficiency of Reservoir Computing to tree-structured domains. By keeping the recursive hidden layer fixed and only training a linear readout, TreeESNs can achieve performance comparable to more complex, fully-trained recursive models.

The TreeESN Architecture

The TreeESN consists of a large hidden layer of recursive units where the weights for the input ($W_{in}$) and recursive connections ($\hat{W}$) are randomly initialized and scaled to ensure stability (contractivity) but are never updated during training. The state of a node is computed based on its own label and the states of its children.

A critical design choice in TreeESNs is state mapping ($\chi$), the process of aggregating the states of an entire tree structure into a single vector for classification or regression. The authors compare two strategies:

  • Root State Mapping: Only the state of the root node is used (the standard approach in RecNNs).
  • Mean State Mapping: The average of all node states in the tree is used.

Experimental Analysis

The model was evaluated on two benchmarks: the Alkanes Dataset (predicting boiling points of molecular structures) and a Sequential Symbolic Task (testing long-term dependencies).

Impact of Mapping and Dynamics

For the Alkanes task, changing the mapping strategy was more impactful than the actual training of recursive weights. Mean state mapping treats every node equally, which better suits chemical properties depending on the whole molecule rather than just a specific "root" atom.

Mean Absolute Error vs. sigma for root and mean state mapping strategies on the Alkanes task.

As shown in the figure above, performance improved as the "contraction coefficient" ($\sigma$) increased toward 1.0, suggesting that richer, less-contractive dynamics are beneficial for these tasks.

Comparative Performance

TreeESNs provide a highly efficient alternative to standard RecNNs. In the Alkanes benchmark, the TreeESN with mean mapping achieved a mean absolute error of 2.60°C, significantly outperforming root state mapping and remaining competitive with state-of-the-art fully trained models.

Comparison of error rates across different recursive models on the Alkanes dataset.

As seen in the table, the TreeESN (mean) performance approaches that of the Neural Network for Graphs (NN4G), which scored 1.74, while avoiding the computational overhead of backpropagation through the tree structure.

Conclusion and Future Work

TreeESNs represent a promising "middle ground" for processing structured data, offering competitive accuracy with vastly reduced training costs. However, the choice between root, mean, or other aggregation methods depends entirely on the nature of the task. Future work will involve testing the model on more complex tree-structured tasks, such as natural language processing, to further explore the limits of fixed-reservoir encoding.

Summary updated

BibTeX
@inproceedings{gallicchio2010treeesn,
  author = {Claudio Gallicchio and Alessio Micheli},
  title = {{TreeESN: a Preliminary Experimental Analysis}},
  booktitle = {ESANN},
  year = {2010},
  url = {https://openreview.net/forum?id=HBGN3ZX_gG},
}
Figure