Fusing Pre-Trained Language Models with Multimodal Prompts through Reinforcement Learning

Youngjae Yu, Jiwan Chung, Heeseung Yun, Jack Hessel, Jae Sung Park, Ximing Lu, Rowan Zellers, Prithviraj Ammanabrolu, Ronan Le Bras, Gunhee Kim, Yejin Choi

Abstract

ESPER (Extending Sensory PErception with Reinforcement learning) is a framework designed to align text-only large language models (LLMs) with multimodal inputs without the need for massive paired datasets. By utilizing CLIP as a reward signal within a reinforcement learning loop, ESPER successfully integrates visual and auditory features into the existing reasoning capabilities of LLMs. This approach achieves a 100x speedup over previous unsupervised methods and establishes new benchmarks for zero-shot multimodal generation and reasoning.

1. Introduction

A significant bottleneck in developing multimodal models is the requirement for vast, curated datasets of paired images and text. While unimodal data is abundant, high-quality paired data for specialized domains—such as ethics, news, or complex commonsense reasoning—is scarce and costly. Furthermore, fine-tuning LLMs on simple image-caption pairs often degrades the rich linguistic and reasoning abilities acquired during text-only pre-training. ESPER addresses this by providing a mechanism to give LLMs "eyes" while preserving their internal "brain," bypassing the need for bespoke supervised datasets.

2. Methodology

ESPER conceptualizes the alignment of vision and language as a reinforcement learning (RL) problem rather than a supervised task. The architecture consists of three primary components: a frozen CLIP model for shared feature encoding, a frozen Pre-trained Language Model (PLM) such as GPT-2, and a small, trainable MLP encoder that maps CLIP features into the LLM’s word embedding space.

The ESPER architecture: (a) Multimodal prompts are encoded; (b) the model performs an online rollout to generate text; (c) a reinforcement learning loop uses CLIP cosine distance as a reward signal.

The system generates text completions based on image prompts and uses CLIP to calculate the cosine similarity between the input image and the generated text. This similarity score serves as a reward signal. Using Proximal Policy Optimization (PPO), the model updates the small encoder to maximize this reward. To maintain linguistic fluency and prevent "reward hacking," a KL-divergence penalty is applied, ensuring the output remains close to the original language model's distribution.

3. Datasets and Evaluation

The framework was evaluated across diverse benchmarks, including MSCOCO for captioning, Visual Commonsense Graph (VCG) and ATOMIC for temporal inference, and VisualNews/VisDial for domain-specific tasks. Additionally, the authors introduced the Evaluation for Styled Prompt (ESP) dataset, containing 1,000 images each paired with five domain-specific captions (blog, social media, instruction, story, and news) to test stylistic adaptation.

4. Key Results

ESPER demonstrates significant improvements in both efficiency and performance:

  • Performance: On MSCOCO, ESPER-CaptionLM achieved a CIDEr score of 78.2 in an unsupervised setting, outperforming previous state-of-the-art methods by nearly 15 points.
  • Zero-Shot Reasoning: In visual commonsense tasks, ESPER performed comparably to supervised models like VisualCOMET, despite not being trained on explicit image-text pairs.
  • Efficiency: ESPER generates outputs in approximately 0.65 seconds, a 100x speedup compared to decoding-time optimization methods like ZeroCap.
Qualitative results from the ESP dataset showing ESPER's ability to adapt its generation style (e.g., Social Media vs. News) based on the provided prompt for the same image.

5. Discussion and Limitations

While ESPER excels in stylistic diversity and efficiency, it faces challenges with specific proper nouns due to CLIP's knowledge limits. It also occasionally exhibits internal inconsistency across different prompts for the same image and struggles with precise spatial logic or counting. Future work may involve more advanced reward models or Reinforcement Learning from Human Feedback (RLHF) to further refine factual accuracy and alignment.

Summary updated

Abstract

Language models are capable of commonsense reasoning: while domain-specific models can learn from explicit knowledge (e.g. commonsense graphs [6] ethical norms [25]), and larger models like GPT-3 [7] mani-fest broad commonsense reasoning capacity. Can their knowledge be extended to multimodal inputs such as images and audio without paired domain data? In this work, we propose ‡ ESPER (Extending Sensory PErception with Reinforcement learning) which enables text-only pretrained models to address multimodal tasks such as visual commonsense reasoning. Our key novelty is to use rein-forcement learning to align multimodal inputs to language model generations without direct supervision: for example, our reward optimization relies only on cosine similarity derived from CLIP [52] and requires no additional paired (image, text) data. Experiments demonstrate that ESPER outperforms baselines and prior work on a variety of multimodal text generation tasks ranging from captioning to commonsense reasoning; these include a new benchmark we collect and release, the ESP dataset, which tasks models with generating the text of several different domains for each image. Our code and data are publicly released at https://github.com/JiwanChung/esper.
BibTeX
@inproceedings{yu2023fusing,
  author = {Youngjae Yu and Jiwan Chung and Heeseung Yun and Jack Hessel and Jae Sung Park and Ximing Lu and Rowan Zellers and Prithviraj Ammanabrolu and Ronan Le Bras and Gunhee Kim and Yejin Choi},
  title = {{Fusing Pre-Trained Language Models with Multimodal Prompts through Reinforcement Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.01044},
}
Figure