Abstract
Visual Program Synthesis enables Large Language Models (LLMs) to solve complex visual tasks by generating executable code that interfaces with specialized vision modules. A significant bottleneck in this field is the absence of large-scale datasets containing ground-truth programs. This paper introduces VisReP (Visually Reinforced Program Synthesis), a method for self-training LLMs using visual reinforcement. By utilizing the final answer of a vision task as a reward signal, VisReP allows models to learn from direct experience without expert-annotated code. We demonstrate that a fine-tuned CodeLlama-7B model can surpass the performance of GPT-3.5-turbo on compositional visual reasoning tasks, showing substantial improvements in accuracy and generalization.
Introduction
While LLMs have demonstrated remarkable capabilities in logical reasoning and language generation, they often struggle with "grounding"—the ability to relate linguistic concepts to specific visual details in an image. Visual Program Synthesis has emerged as a powerful paradigm to bridge this gap. In this approach, an LLM acts as a high-level controller, writing Python programs that invoke specialized tools (e.g., object detectors, depth estimators) to decompose a complex query into executable steps.
Despite its promise, current state-of-the-art implementations typically rely on "frozen" proprietary models like GPT-4 through few-shot prompting. Training these models specifically for visual programming is difficult because high-quality, ground-truth programs are not readily available in large quantities. Crowdsourcing such data is prohibitively expensive and requires specialized technical knowledge. The central challenge addressed here is: How can we train an LLM to write better visual programs using only the feedback from the execution of those programs?
The VisReP Framework
The authors propose VisReP, which treats the LLM as a policy within a reinforcement learning framework. The core innovation is the use of "visual reinforcement" to close the loop between code generation and task success.
1. The Reward Signal
Instead of requiring human-graded code, VisReP leverages existing vision-language annotations (e.g., the "Yes/No" or short-form answers in VQA datasets). The system executes the generated program; if the output matches the ground-truth answer, the program receives a reward of 1. Otherwise, it receives a reward of 0. This "coarse" reward signal bypasses the need for intermediate program supervision.
2. Iterative Self-Training
The training process consists of two primary phases:
- Synthetic Data Generation (Grow Step): The LLM policy generates multiple candidate programs for a diverse set of visual queries.
- Offline Policy Improvement (Improve Step): The system filters these candidates, selecting only the programs that successfully produced the correct answer. The LLM is then fine-tuned on this successful subset using Low-Rank Adaptation (LoRA), effectively imitating its own successful behaviors.
3. Stabilization Strategies
A known risk in self-training is "learning the right thing for the wrong reasons"—for instance, a buggy program might coincidentally produce the correct "Yes" answer for a binary question. To mitigate this, the authors identify "hard" question types where accuracy is low and introduce a minimal amount of human-corrected code as in-context examples to guide the model's exploration.
Experimental Evaluation
VisReP was evaluated across several challenging benchmarks:
- GQA: A dataset focused on compositional Visual Question Answering.
- Omnilabel: Complex object detection tasks involving long, free-form descriptions.
- SugarCrepe & WinoGround: Benchmarks for image-text matching that test spatial and logical understanding.
The researchers also tested generalization on VQAv2 and OK-VQA to ensure the model learned transferable skills rather than dataset-specific shortcuts.
Key Results and Insights
- Performance Gains: On GQA, VisReP improved the base CodeLlama model's accuracy by 9%, reaching 59.2% and outperforming GPT-3.5-turbo (53.9%).
- Efficiency and Generalization: The method proved robust even with limited data. Furthermore, a model trained on VQA tasks (GQA) showed improved performance on object detection (Omnilabel), indicating the acquisition of general "visual programming" capabilities.
- Structural Convergence: Analysis of Abstract Syntax Trees (ASTs) revealed that as training progressed, the model's code generation became more consistent, "honing in" on effective programmatic structures rather than exploring random variations.
- The Binary Challenge: The authors noted that self-training is particularly sensitive to binary (Yes/No) questions, where the 50% chance of a lucky guess can reinforce incorrect logic. This underscores the importance of stabilization techniques.
Conclusion
VisReP demonstrates that LLMs can effectively teach themselves to solve complex visual tasks through a self-reinforcing loop of code generation and execution. By shifting the supervisory signal from "correct code" to "correct results," the framework enables the specialization of smaller, open-source models to a level that rivals or exceeds much larger proprietary systems. Future work will explore more granular reward models to provide step-by-step feedback, further refining the logical consistency of synthesized programs.
Summary updated