TL;DR
This paper introduces ESPER, a novel framework that extends the zero-shot capabilities of large language models (LLMs) to multimodal inputs like images and audio. ESPER uses reinforcement learning to align multimodal information with LLM generations without requiring explicitly paired data, enabling LLMs to perform diverse tasks like image captioning in various styles.
What Problem Does This Paper Address?
Large language models (LLMs) have shown remarkable zero-shot generalization abilities for text-based tasks. However, their input is typically limited to text. The challenge addressed here is how to extend this zero-shot capacity to multimodal inputs (images, audio) for a wide range of generation tasks, without needing extensive task-specific, paired multimodal datasets. Existing methods often rely on maximum likelihood estimation with paired data, which can limit the diversity and richness of generated outputs.
Methods
ESPER combines a frozen, pre-trained language model (like GPT-2) with a lightweight, trainable multimodal encoder. The core innovation is the use of reinforcement learning (RL) for alignment. Instead of direct supervision, ESPER uses a reward signal derived from a pre-trained multimodal model (CLIP) to guide the training of the encoder. Specifically:
- Multimodal Encoder: A small neural network that maps multimodal inputs (e.g., image features from CLIP) into the LLM's embedding space. This acts as a "multimodal prompt."
- Reinforcement Learning: Proximal Policy Optimization (PPO) is used to train the encoder. The reward function is the cosine similarity between the CLIP-encoded input and the CLIP-encoded text generated by the LLM. This encourages the LLM to produce text that is semantically aligned with the input modality.
- Frozen LLM: The parameters of the LLM are kept frozen, preserving its original zero-shot generalization capabilities.
- Style Conditioning (ESPER-Style): For enhanced stylistic control, the LLM backbone can be optionally fine-tuned on text-only corpora with specific style prefixes (e.g., "news:", "story:").
Datasets
- COCO Captions: Used for training the multimodal encoder and for evaluating unpaired image captioning.
- AudioCaps: Used for evaluating unpaired audio captioning.
- ESP Dataset (New Benchmark): Introduced by the authors, this dataset consists of 4.8k captions from 1k COCO images, featuring five diverse writing styles (blog, social media, instruction, story, news). It's designed to test a model's ability to generate styled captions for the same image without explicit visual-text pairing.
- VisualNews: Used for evaluating visual news generation.
- VisDial: Used for evaluating visual dialogue generation.
Key Insights and Ideas
- RL for Multimodal Alignment: Using RL with a CLIP-derived reward is a powerful way to align multimodal inputs with LLM generations without needing explicit paired data. This allows for more flexible and diverse generation.
- Preserving LLM Generalization: Keeping the LLM parameters frozen is crucial. It ensures that the model retains its broad zero-shot capabilities while learning to interpret new modalities.
- Style Transfer via Prompts: Pre-training the LLM with style prefixes (ESPER-Style) effectively imbues it with the ability to generate text in specific styles, demonstrating that style can be controlled through text prompts.
- Beyond Literal Captions: ESPER can generate more than just literal descriptions; it can produce creative and contextually relevant text for various prompts and styles, as shown in its performance on the ESP dataset.
Key Results
- Unpaired Image Captioning: ESPER outperforms several state-of-the-art unsupervised methods on COCO test split, achieving higher CIDEr scores. ESPER-Style, with style pre-training, further improves performance.
- Audio Captioning: ESPER achieves better results than baseline models on the AudioCaps dataset, demonstrating its ability to handle audio inputs.
- Diverse Style Generation (ESP Dataset): ESPER-Style significantly outperforms baselines like CLIPCap-MLP and text-only models on the new ESP dataset, showcasing its ability to generate diverse styles conditioned on images.
- Visual News and Dialogue: ESPER shows strong performance on VisualNews and VisDial, even outperforming supervised baselines in some cases, highlighting its generalization capabilities.
- Efficiency: ESPER's inference speed is comparable to vanilla GPT-2, with only a lightweight encoder adding overhead, making it significantly faster than methods like CLIP-Infer that rely on per-token gradient optimization.
Limitations and Future Work
- Audio Encoder Limitations: The accuracy of audio-based rewards from models like Wav2CLIP might be less precise compared to image-based CLIP, potentially due to smaller, less diverse audio training datasets.
- Knowledge Gaps: For specific domains like news, CLIP might have knowledge gaps (e.g., proper nouns), which can affect performance metrics like CIDEr. Fine-tuning can help bridge these gaps.
- Future Work:
- Extending ESPER to simultaneously maximize rewards for multiple modalities (image, audio, OCR, etc.).
- Scaling up the CLIP and GPT-2 backbones to larger variants.
- Exploring ESPER's utility as a data augmentation tool for multimodal reasoning tasks.
Summary updated