Recurrent Nested Model for Sequence Generation

Wenhao Jiang, Lin Ma, Wei Lu

Abstract

The authors propose a Recurrent Nested Model (RNM) that significantly increases the depth of encoder-decoder architectures by inserting a multi-layered "reviewer" module between them. By using a nested structure with dense inter-block and intra-block connections, the model successfully scales the "review" phase from 8 steps to 64, achieving state-of-the-art results in image and code captioning tasks. This architecture effectively overcomes the training difficulties and performance plateaus previously associated with deep recurrent review processes.

Introduction

In sequence generation tasks, such as image captioning or code summarization, the standard approach utilizes an encoder to process input and a decoder to generate text. While previous research, notably the "Review Net," introduced intermediate "review steps" to refine input representations, these models encountered a "depth ceiling." Performance typically dropped when review depth exceeded 8 steps due to vanishing gradients and information loss. The Recurrent Nested Model (RNM) is designed to break this barrier, allowing for much deeper intermediate "thought" processes that remain effective and trainable.

Methods

The core innovation of the RNM is the Reviewer Module, which is positioned between the encoder and decoder. This module is organized into a nested hierarchy to facilitate deep learning without degradation:

  1. Nested Blocks: The reviewer is divided into $B$ blocks, each containing $L$ recurrent transition steps (using LSTMs).
  2. Intra-block Connections: Within a single block, each step is connected to all previous steps in that same block, creating a "higher-order RNN" that captures complex local dependencies.
  3. Inter-block Connections: Each block receives refined representations (thought vectors) from every preceding block, ensuring that information from early stages is preserved throughout the deep architecture.
  4. Discriminative Supervision: An auxiliary loss function forces intermediate thought vectors to predict frequent target words, guiding the hidden states toward useful representations.
Overview of the Recurrent Nested Model architecture showing the nested reviewer blocks and dense connectivity.

Key Results and Insights

The model was evaluated on the MS COCO dataset for image captioning and the HabeasCorpus dataset for code captioning.

Breaking the Depth Barrier

A primary finding is that while previous models plateaued at a review depth of 8, the RNM continues to improve up to a depth of 64 (8 blocks of 8 steps). The combination of local (intra) and global (inter) connections allows gradients to flow back to early layers, preventing the vanishing gradient problem.

Comparison of CIDEr scores across different review depths, demonstrating how the RNM outperforms the Review Net as depth increases.

Performance Metrics

  • Image Captioning: On the MS COCO test set, the RNM achieved a CIDEr score of 109.7 (single model) and 113.1 (ensemble), outperforming the original Review Net and other competitive baselines like Att2all.
  • Code Captioning: The model showed a clear advantage in "Character Savings" (CS-k). At CS-1 (predicting the exact next word), it outperformed standard LSTMs and soft-attention models, proving the architecture's versatility across different encoder types.
  • Ablation Studies: Removing either inter-block or intra-block connections led to a measurable drop in CIDEr scores, confirming that both types of dense connections are necessary for the model's success.

Conclusion

The Recurrent Nested Model demonstrates that depth in the review phase is highly beneficial for sequence generation when supported by a specific nested topology. While increasing depth to 64 steps introduces computational overhead, the resulting performance gains are substantial. Future work may involve applying this nested structure to non-recurrent architectures, such as Transformers, to explore similar benefits in depth and information preservation.

Summary updated

Abstract

Depth has been shown beneficial to neural network models. In this paper, we make an attempt to make the encoder-decoder model deeper for sequence generation. We propose a module that can be plugged into the middle between the encoder and decoder to increase the depth of the whole model. The proposed module follows a nested structure, which is divided into blocks with each block containing several recurrent transition steps. To reduce the training difficulty and preserve the necessary information for the decoder during transitions, inter-block connections and intra-block connections are constructed in our model. The inter-block connections provide the thought vectors from the current block to all the subsequent blocks. The intra-block connections connect all the hidden states entering the current block to the current transition step. The advantages of our model are illustrated on the image captioning and code captioning tasks.
BibTeX
@inproceedings{jiang2020recurrent,
  author = {Wenhao Jiang and Lin Ma and Wei Lu},
  title = {{Recurrent Nested Model for Sequence Generation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i07.6768},
}
Figure