A Chain-of-Thought Subspace Meta-Learning for Few-shot Image Captioning with Large Vision and Language Models

Hao Huang, Shuaihang Yuan, Yu Hao, Congcong Wen, Yi FangarXiv 2025 · DOI · arXiv

Abstract

This paper introduces a Chain-of-Thought (CoT) Subspace Meta-Learning framework designed to improve image captioning when only a few training examples are available. By breaking the captioning process into logical steps (Subject $\rightarrow$ Object $\rightarrow$ Full Caption) and learning these steps in distinct mathematical subspaces, the framework successfully bridges the gap between frozen Large Vision Models (LVMs) and Large Language Models (LLMs), outperforming existing few-shot baselines.

Introduction

Image captioning requires a model to understand visual features and translate them into coherent natural language. While Large Vision Models (like CLIP) and Large Language Models (like GPT) are powerful, connecting them is difficult—especially in few-shot settings where the model must adapt to new types of images with only 1 to 5 examples.

Existing methods often use a "one-step" prompting approach, where a single visual feature is fed into an LLM. The authors argue this is insufficient for complex scenes. Humans identify subjects, notice objects, and then reason about their interactions. Furthermore, standard meta-learning (like MAML) tries to learn all these different reasoning steps in a single parameter space, which can lead to "interference" where learning how to identify one element negatively impacts the model's ability to describe another.

Methodology

The authors propose a two-pronged solution to these challenges:

  1. Chain-of-Thought (CoT) Prompting: Instead of generating a caption in one go, the model follows a sequential reasoning path based on the Subject-Verb-Object (SVO) structure. It first identifies the subject, then the object, and finally uses these as "thought" prompts to generate the full descriptive sentence.
The Chain-of-Thought (CoT) captioning process breaks down the task into sequential reasoning steps: identifying the subject, then the object, and finally generating the full caption.
  1. Subspace Meta-Learning: To prevent different reasoning steps from interfering with each other, the model learns meta-parameters within distinct subspaces. Each step of the CoT (Subject, Object, Caption) has its own subspace defined by a basis matrix. The model learns how to combine these bases to adapt quickly to new tasks.

Architecture

The system utilizes a frozen CLIP (ViT-B/32) vision encoder and a frozen GPT-2 language model. The core innovation lies in the Meta-Adaptor, a lightweight, learnable multi-head attention block that transforms visual features into "soft prompts" that the LLM can understand. Only these adaptors are trained using a meta-learning algorithm (MAML).

The proposed architecture utilizes frozen vision and language models connected by learnable meta-adaptors that operate within distinct subspaces for each reasoning step.

Evaluation and Key Results

The researchers evaluated their model on three gold-standard image captioning benchmarks: MSCOCO-2017, Flickr8k, and Flickr30k. The experiments were conducted in "N-way K-shot" settings to test the model's ability to learn from very few examples of specific object categories.

Key findings include:

  • Superior Accuracy: In 5-way 5-shot settings on MSCOCO, the model achieved a CIDEr score of 80.61, significantly higher than the Meta-Mapper baseline (69.06).
  • Cross-Domain Robustness: The model showed superior performance when trained on MSCOCO and tested on Flickr30k, indicating that the CoT subspace strategy helps the model learn generalizable reasoning patterns rather than dataset-specific styles.
  • Mimicking Human Cognition: By forcing the model to "think" about the subject and object separately, the final caption becomes more grounded in the actual image content.

Conclusion and Future Work

By isolating the parameters for different reasoning steps into subspaces, the model avoids the interference issues common in standard gradient-based meta-learning. While the model currently relies on the LLM's internal "language prior" to handle verbs, future work could explore explicitly incorporating action recognition and scaling the framework to larger foundation models like Llama 3.

Summary updated

Abstract

A large-scale vision and language model that has been pretrained on massive data encodes visual and linguistic prior, which makes it easier to generate images and language that are more natural and realistic. Despite this, there is still a significant domain gap between the modalities of vision and language, especially when training data is scarce in few-shot settings, where only very limited data are available for training. In order to mitigate this issue, a multi-modal meta-learning framework has been proposed to bridge the gap between two frozen pretrained large vision and language models by introducing a tunable prompt connecting these two large models. For few-shot image captioning, the existing multi-model meta-learning framework utilizes a one-step prompting scheme to accumulate the visual features of input images to guide the language model, which struggles to generate accurate image descriptions with only a few training samples. Instead, we propose a chain-of-thought (CoT) meta-learning scheme as a multi-step image captioning procedure to better imitate how humans describe images. In addition, we further propose to learn different meta-parameters of the model corresponding to each CoT step in distinct subspaces to avoid interference. We evaluated our method on three commonly used image captioning datasets, i.e., MSCOCO, Flickr8k, and Flickr30k, under few-shot settings. The results of our experiments indicate that our chain-of-thought subspace meta-learning strategy is superior to the baselines in terms of performance across different datasets measured by different metrics.
BibTeX
@misc{huang2025chain,
  author = {Hao Huang and Shuaihang Yuan and Yu Hao and Congcong Wen and Yi Fang},
  title = {{A Chain-of-Thought Subspace Meta-Learning for Few-shot Image Captioning with Large Vision and Language Models}},
  year = {2025},
  eprint = {2502.13942},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.13942},
}
Figure