Abstract
The authors propose a Recurrent Nested Model (RNM) that significantly increases the depth of encoder-decoder architectures by inserting a multi-layered "reviewer" module between them. By using a nested structure with dense inter-block and intra-block connections, the model successfully scales the "review" phase from 8 steps to 64, achieving state-of-the-art results in image and code captioning tasks. This architecture effectively overcomes the training difficulties and performance plateaus previously associated with deep recurrent review processes.
Introduction
In sequence generation tasks, such as image captioning or code summarization, the standard approach utilizes an encoder to process input and a decoder to generate text. While previous research, notably the "Review Net," introduced intermediate "review steps" to refine input representations, these models encountered a "depth ceiling." Performance typically dropped when review depth exceeded 8 steps due to vanishing gradients and information loss. The Recurrent Nested Model (RNM) is designed to break this barrier, allowing for much deeper intermediate "thought" processes that remain effective and trainable.
Methods
The core innovation of the RNM is the Reviewer Module, which is positioned between the encoder and decoder. This module is organized into a nested hierarchy to facilitate deep learning without degradation:
- Nested Blocks: The reviewer is divided into $B$ blocks, each containing $L$ recurrent transition steps (using LSTMs).
- Intra-block Connections: Within a single block, each step is connected to all previous steps in that same block, creating a "higher-order RNN" that captures complex local dependencies.
- Inter-block Connections: Each block receives refined representations (thought vectors) from every preceding block, ensuring that information from early stages is preserved throughout the deep architecture.
- Discriminative Supervision: An auxiliary loss function forces intermediate thought vectors to predict frequent target words, guiding the hidden states toward useful representations.
Key Results and Insights
The model was evaluated on the MS COCO dataset for image captioning and the HabeasCorpus dataset for code captioning.
Breaking the Depth Barrier
A primary finding is that while previous models plateaued at a review depth of 8, the RNM continues to improve up to a depth of 64 (8 blocks of 8 steps). The combination of local (intra) and global (inter) connections allows gradients to flow back to early layers, preventing the vanishing gradient problem.
Performance Metrics
- Image Captioning: On the MS COCO test set, the RNM achieved a CIDEr score of 109.7 (single model) and 113.1 (ensemble), outperforming the original Review Net and other competitive baselines like Att2all.
- Code Captioning: The model showed a clear advantage in "Character Savings" (CS-k). At CS-1 (predicting the exact next word), it outperformed standard LSTMs and soft-attention models, proving the architecture's versatility across different encoder types.
- Ablation Studies: Removing either inter-block or intra-block connections led to a measurable drop in CIDEr scores, confirming that both types of dense connections are necessary for the model's success.
Conclusion
The Recurrent Nested Model demonstrates that depth in the review phase is highly beneficial for sequence generation when supported by a specific nested topology. While increasing depth to 64 steps introduces computational overhead, the resulting performance gains are substantial. Future work may involve applying this nested structure to non-recurrent architectures, such as Transformers, to explore similar benefits in depth and information preservation.
Summary updated