Abstract
Traditional deep learning-based image captioning models are "closed-book" systems that store all knowledge within fixed parameters. This leads to rigidity, a lack of interpretability, and limitations in retrieval granularity. This paper introduces a "plug-and-play" memory-augmented (MA) mechanism that allows models to look up word-level knowledge from an external database during inference. By linearly interpolating a standard model's predictions with retrieved information from a non-parametric memory bank, the authors achieve state-of-the-art results on the MS COCO benchmark without requiring additional training parameters.
Introduction
Conventional image captioning models typically rely on internal parameters to generate descriptions, making them difficult to update or interpret. When these models attempt to incorporate external knowledge, they often use sentence-level retrieval, which can lead to incoherent captions if the retrieved sentence is even slightly irrelevant. To address these issues, this research proposes a shift toward word-level retrieval. By consulting a "dictionary" of past experiences for every word generated, the model can dynamically adapt its output based on similar historical contexts found in the training data.
Methodology
The proposed system enhances a standard image captioning (IC) model with a non-parametric memory bank and a retrieval-interpolation mechanism. During the inference process, the model does not rely solely on its internal weights; instead, it queries an external database to refine its predictions.
Memory Bank Construction
The memory bank is an offline database of Key-Value pairs.
- Key: A vector representation of the "history context," which combines image features with the sequence of words generated up to the current time step.
- Value: The actual ground-truth word that followed that specific context in the training set.
Word-Level Retrieval and Combined Inference
At each time step $t$ during inference, the model encodes the current context into a query vector. It then uses the FAISS library for efficient similarity search to identify the top-$k$ most similar keys in the memory bank.
The final word prediction is determined through Combined Inference. The model calculates a probability distribution from the retrieved matches ($P_{MA}$) and blends it with the base model’s predicted distribution ($P_{IC}$) using a balancing parameter $\lambda$. Because the memory bank is non-parametric and utilizes the model's existing hidden states, this augmentation requires zero additional training parameters.
Key Insights and Results
The researchers evaluated the method on the MS COCO dataset using the standard Karpathy split. Several key insights emerged from the study:
- Granularity and Robustness: Retrieving at the word level allows the model to "mix and match" knowledge from various training examples, making it significantly more robust to noisy or irrelevant retrieval results compared to sentence-level methods.
- State-of-the-Art Performance: The memory-augmented versions consistently outperformed baselines across all metrics. For instance, an AoANet model equipped with MA achieved a CIDEr score of 132.0, compared to the base model's 129.8.
- Human Evaluation: In head-to-head tests, humans judged the MA-enhanced captions as superior in terms of richness (38.4% wins vs. 21.4% for the baseline).
- Efficient Knowledge Transfer: A model trained on a small dataset can be improved simply by providing it with a memory bank derived from a larger dataset, facilitating knowledge transfer without gradient updates.
Limitations and Future Work
Despite its performance gains, the approach introduces a computational overhead, making inference approximately 3x slower than the base model. Additionally, storing millions of key-value pairs requires a significant memory footprint, though this is partially mitigated by FAISS quantization. Future research may explore cross-modality knowledge construction to better integrate visual and textual information, as well as applying this framework to other tasks like machine translation or video captioning.
Summary updated