Abstract
ESPER (Extending Sensory PErception with Reinforcement learning) is a framework designed to align text-only large language models (LLMs) with multimodal inputs without the need for massive paired datasets. By utilizing CLIP as a reward signal within a reinforcement learning loop, ESPER successfully integrates visual and auditory features into the existing reasoning capabilities of LLMs. This approach achieves a 100x speedup over previous unsupervised methods and establishes new benchmarks for zero-shot multimodal generation and reasoning.
1. Introduction
A significant bottleneck in developing multimodal models is the requirement for vast, curated datasets of paired images and text. While unimodal data is abundant, high-quality paired data for specialized domains—such as ethics, news, or complex commonsense reasoning—is scarce and costly. Furthermore, fine-tuning LLMs on simple image-caption pairs often degrades the rich linguistic and reasoning abilities acquired during text-only pre-training. ESPER addresses this by providing a mechanism to give LLMs "eyes" while preserving their internal "brain," bypassing the need for bespoke supervised datasets.
2. Methodology
ESPER conceptualizes the alignment of vision and language as a reinforcement learning (RL) problem rather than a supervised task. The architecture consists of three primary components: a frozen CLIP model for shared feature encoding, a frozen Pre-trained Language Model (PLM) such as GPT-2, and a small, trainable MLP encoder that maps CLIP features into the LLM’s word embedding space.
The system generates text completions based on image prompts and uses CLIP to calculate the cosine similarity between the input image and the generated text. This similarity score serves as a reward signal. Using Proximal Policy Optimization (PPO), the model updates the small encoder to maximize this reward. To maintain linguistic fluency and prevent "reward hacking," a KL-divergence penalty is applied, ensuring the output remains close to the original language model's distribution.
3. Datasets and Evaluation
The framework was evaluated across diverse benchmarks, including MSCOCO for captioning, Visual Commonsense Graph (VCG) and ATOMIC for temporal inference, and VisualNews/VisDial for domain-specific tasks. Additionally, the authors introduced the Evaluation for Styled Prompt (ESP) dataset, containing 1,000 images each paired with five domain-specific captions (blog, social media, instruction, story, and news) to test stylistic adaptation.
4. Key Results
ESPER demonstrates significant improvements in both efficiency and performance:
- Performance: On MSCOCO, ESPER-CaptionLM achieved a CIDEr score of 78.2 in an unsupervised setting, outperforming previous state-of-the-art methods by nearly 15 points.
- Zero-Shot Reasoning: In visual commonsense tasks, ESPER performed comparably to supervised models like VisualCOMET, despite not being trained on explicit image-text pairs.
- Efficiency: ESPER generates outputs in approximately 0.65 seconds, a 100x speedup compared to decoding-time optimization methods like ZeroCap.
5. Discussion and Limitations
While ESPER excels in stylistic diversity and efficiency, it faces challenges with specific proper nouns due to CLIP's knowledge limits. It also occasionally exhibits internal inconsistency across different prompts for the same image and struggles with precise spatial logic or counting. Future work may involve more advanced reward models or Reinforcement Learning from Human Feedback (RLHF) to further refine factual accuracy and alignment.
Summary updated