TL;DR
CoCoEvolve introduces an annotation-free, self-supervised framework that enforces semantic consistency across chart images, data tables, and rendering code. By leveraging mutual agreement as an optimization signal during both training and test-time inference, it significantly improves cross-modal understanding and eliminates the need for expensive, ambiguous one-to-many annotations.
What Problem Does This Paper Address?
Data visualizations exist in three primary forms: visual chart images, structured data tables, and declarative rendering code (like matplotlib scripts). For AI systems to truly understand charts, they must be able to fluidly translate between all three modalities.
However, current multimodal large language models (MLLMs) struggle with this because the relationships between these representations are inherently one-to-many. For instance, a single chart image can be faithfully represented by multiple variations of data tables or entirely different code structures. Traditional supervised learning tries to force these into rigid, one-to-one ground truths, which makes data annotation incredibly expensive and ambiguous. Furthermore, training models on isolated tasks (like chart-to-table) often degrades their performance on reversed or unseen tasks (like table-to-code), leading to fragmented understanding and highly inconsistent outputs.
Methods
The authors introduce CoCoEvolve, a consistency-driven co-evolution framework that treats charts ($V$), tables ($T$), and code ($C$) as equal representations of the same underlying semantics. Instead of relying on human labels, it uses a closed representation cycle ($V \rightarrow T \rightarrow C \rightarrow V$) to let two models—a multimodal model $M_\theta$ and a language model $M_\psi$—supervise each other.
The framework operates in three distinct phases:
- CoCoEvolve@Train: The models are jointly optimized over unannotated chart images using Reinforcement Learning (supporting algorithms like GRPO, DAPO, and GSPO). It employs a hierarchical reward system:
- Code Consistency Reward: Measures if the generated code executes successfully and matches semantically via embedding similarity.
- Visual Consistency Reward: Uses vision models (CLIP, DINOv2, SSIM, OCR) to check if the code-rendered chart matches the original input chart.
- Format Reward: Enforces well-formed structured outputs.
- CoCoEvolve@Test: At inference time, the same consistency objectives are used to evaluate multiple candidate outputs (rollouts) and select the most coherent result, providing test-time optimization without any ground truth.
- CoCoEvolve@Eval: To fix the unreliability of standard LLM-as-judge scoring (which collapses complex visual fidelity into a single 0-100 score), the authors propose a multi-dimensional evaluation suite that decomposes chart, code, table, and constraint accuracy into fine-grained, interpretable dimensions.
Datasets
The framework was trained and evaluated across four distinct benchmarks:
- ChartCoder: The primary source for training (10,298 filtered instances) and in-domain evaluation (542 instances).
- ChartNet: A million-scale dataset used here for out-of-domain evaluation across diverse topics (e.g., health, finance).
- ChartMimic: Used to evaluate multi-domain chart reproduction and modification tasks across scientific disciplines like physics and biology.
- Chart2Code: Used to test model robustness across varying levels of chart complexity and multi-input modification instructions.
Key Insights and Ideas
- Consistency as a Free Signal: If multiple modalities describe the same data, they must mutually agree. This agreement acts as a powerful, annotation-free self-supervision signal that bypasses the one-to-many ambiguity problem.
- Error Propagation: The authors discovered that model failures are highly correlated across representations. An error in table extraction directly propagates into corrupted code generation and distorted visual rendering. Enforcing cycle consistency effectively breaks this chain of failure.
- Anti-Collusion via Asymmetric Weights: If two models only optimize for mutual agreement, they might "collude" to produce matching but entirely incorrect outputs. To prevent this reward hacking, CoCoEvolve uses asymmetric weighting: $M_\theta$ is heavily penalized if it deviates from the original chart image, anchoring the entire cycle to the ground-truth visual input.
Key Results
- Perfect Execution Success: When combining training and test-time optimization (
@Train + @Test), models like Qwen3-8B and Llama3.2-3B achieved a 100% sandbox execution success rate on code generation tasks, up from baselines as low as 22.5%. - Substantial Quantitative Gains: CoCoEvolve yielded up to a 37.91% improvement on non-overlapping test sets for chart-to-code reproduction and up to 46.88% on complex out-of-domain chart modifications.
- Reduced Computational Overhead: By using test-time consistency signals to select the best candidate rollouts, CoCoEvolve actually reduced API evaluation costs substantially (e.g., dropping from \$0.8637 to \$0.2382 per sample compared to sequential baseline generation).
Limitations and Future Work
- Model Scale Constraints: The framework was primarily validated on smaller-scale model architectures (ranging from 1.3B to 8B parameters). The authors note that exploring how co-evolution dynamics scale with massive models (e.g., 480B+) remains an open question.
- Static Images Only: Currently, CoCoEvolve only operates on static chart images. Extending the framework to handle dynamic or animated visualizations—where temporal semantics across frames introduce extra complexity—is highlighted as an exciting future direction.
Summary updated