Introduction
Catastrophic forgetting remains a primary hurdle in neural network research, where models trained on a sequence of tasks tend to overwrite weights essential for previous knowledge. While significant progress has been made in feed-forward networks for image classification, recurrent models used for sequential generation tasks like image captioning have remained underexplored.
Image captioning presents a unique challenge: transient vocabularies. Unlike classification tasks with disjoint labels, captioning tasks share common words (e.g., "a", "the", "is") while specific nouns appear and disappear across tasks. This overlapping nature, combined with the way recurrent connections in LSTMs amplify small weight changes, makes standard continual learning (CL) techniques less effective. This paper introduces Recurrent Attention to Transient Tasks (RATT) to address these specific challenges.
Methodology
RATT is an attention-masking technique designed to protect knowledge within LSTMs by allocating specific subsets of neurons to specific tasks.
Key Components of RATT:
- Attention Masks: The model learns task-conditioned masks (values between 0 and 1) applied to both word embeddings ($a_x$) and LSTM hidden states ($a_h$). These act as gates that define which neurons are active for a given task.
- Selective Backpropagation: To prevent forgetting, RATT utilizes a cumulative "backward mask." If a neuron was critical for a previous task, the backward mask inhibits gradient updates to its weights during subsequent training.
- Growing Vocabulary: Instead of separate heads, RATT uses a single, incrementally growing word embedding and classifier matrix. It deactivates weights for words not present in the current task's vocabulary, efficiently handling the transient nature of language.
Experimental Setup
To evaluate RATT, the authors established two new benchmarks for continual image captioning:
- MS-COCO (Disjoint Visual Categories): Five tasks split by super-categories (transport, animals, sports, food, interior).
- Flickr30k (Incremental Visual Categories): Four tasks (scene, animals, vehicles, instruments) where visual concepts from early tasks can reappear later, providing a more realistic and "messy" CL scenario.
Key Results
RATT demonstrates a significant advantage over standard CL baselines like Elastic Weight Consolidation (EWC) and Learning without Forgetting (LwF).
On the MS-COCO benchmark, RATT achieved a 0.0% forgetting rate across all five tasks, whereas the fine-tuning baseline lost nearly 60% of its performance. On the more complex Flickr30k dataset, RATT even exhibited slight backward transfer (-2.5% forgetting), meaning it improved its performance on older tasks as it learned new ones.
Ablation studies highlighted that while masking hidden states is beneficial, masking word embeddings is the most critical factor in preventing catastrophic forgetting in recurrent architectures.
Discussion and Limitations
While RATT excels at stability, it faces a trade-off with plasticity. By freezing neurons to protect old knowledge, the model eventually has fewer free parameters to learn new tasks. Furthermore, while it prevents forgetting, it does not inherently facilitate strong backward transfer in all scenarios.
Future research may involve applying these transient task masking principles to more modern architectures, such as Transformer-based image captioning systems, to see if the stability-plasticity balance can be further optimized.
Summary updated