Consistency-Driven Co-Evolution for Self-Supervised Cross-Representation Learning

Xuehang Guo, Pengyuan Li, Tom Hope, Tirthankar Ghosal, Manling Li, Qingyun WangarXiv 2026 · arXiv

TL;DR

CoCoEvolve introduces an annotation-free, self-supervised framework that enforces semantic consistency across chart images, data tables, and rendering code. By leveraging mutual agreement as an optimization signal during both training and test-time inference, it significantly improves cross-modal understanding and eliminates the need for expensive, ambiguous one-to-many annotations.

What Problem Does This Paper Address?

Data visualizations exist in three primary forms: visual chart images, structured data tables, and declarative rendering code (like matplotlib scripts). For AI systems to truly understand charts, they must be able to fluidly translate between all three modalities.

However, current multimodal large language models (MLLMs) struggle with this because the relationships between these representations are inherently one-to-many. For instance, a single chart image can be faithfully represented by multiple variations of data tables or entirely different code structures. Traditional supervised learning tries to force these into rigid, one-to-one ground truths, which makes data annotation incredibly expensive and ambiguous. Furthermore, training models on isolated tasks (like chart-to-table) often degrades their performance on reversed or unseen tasks (like table-to-code), leading to fragmented understanding and highly inconsistent outputs.

Figure 1: Chart-Table-Code Representation Cy- cle. Cross-representation understanding across six tasks poses great challenges for model learning and optimization due to its inherent one-to-many nature, supervision ambiguity, and high computa- tional cost (§1).

Methods

The authors introduce CoCoEvolve, a consistency-driven co-evolution framework that treats charts ($V$), tables ($T$), and code ($C$) as equal representations of the same underlying semantics. Instead of relying on human labels, it uses a closed representation cycle ($V \rightarrow T \rightarrow C \rightarrow V$) to let two models—a multimodal model $M_\theta$ and a language model $M_\psi$—supervise each other.

The framework operates in three distinct phases:

  1. CoCoEvolve@Train: The models are jointly optimized over unannotated chart images using Reinforcement Learning (supporting algorithms like GRPO, DAPO, and GSPO). It employs a hierarchical reward system:
  2. Code Consistency Reward: Measures if the generated code executes successfully and matches semantically via embedding similarity.
  3. Visual Consistency Reward: Uses vision models (CLIP, DINOv2, SSIM, OCR) to check if the code-rendered chart matches the original input chart.
  4. Format Reward: Enforces well-formed structured outputs.
  5. CoCoEvolve@Test: At inference time, the same consistency objectives are used to evaluate multiple candidate outputs (rollouts) and select the most coherent result, providing test-time optimization without any ground truth.
  6. CoCoEvolve@Eval: To fix the unreliability of standard LLM-as-judge scoring (which collapses complex visual fidelity into a single 0-100 score), the authors propose a multi-dimensional evaluation suite that decomposes chart, code, table, and constraint accuracy into fine-grained, interpretable dimensions.
Figure 2: CoCoEvolve Overview. We introduce the consistency-driven co-evolution framework. Dashed arrows (- - -) denote gradient updates at training time; models are frozen at test time.

Datasets

The framework was trained and evaluated across four distinct benchmarks:

  • ChartCoder: The primary source for training (10,298 filtered instances) and in-domain evaluation (542 instances).
  • ChartNet: A million-scale dataset used here for out-of-domain evaluation across diverse topics (e.g., health, finance).
  • ChartMimic: Used to evaluate multi-domain chart reproduction and modification tasks across scientific disciplines like physics and biology.
  • Chart2Code: Used to test model robustness across varying levels of chart complexity and multi-input modification instructions.

Key Insights and Ideas

  • Consistency as a Free Signal: If multiple modalities describe the same data, they must mutually agree. This agreement acts as a powerful, annotation-free self-supervision signal that bypasses the one-to-many ambiguity problem.
  • Error Propagation: The authors discovered that model failures are highly correlated across representations. An error in table extraction directly propagates into corrupted code generation and distorted visual rendering. Enforcing cycle consistency effectively breaks this chain of failure.
  • Anti-Collusion via Asymmetric Weights: If two models only optimize for mutual agreement, they might "collude" to produce matching but entirely incorrect outputs. To prevent this reward hacking, CoCoEvolve uses asymmetric weighting: $M_\theta$ is heavily penalized if it deviates from the original chart image, anchoring the entire cycle to the ground-truth visual input.

Key Results

  • Perfect Execution Success: When combining training and test-time optimization (@Train + @Test), models like Qwen3-8B and Llama3.2-3B achieved a 100% sandbox execution success rate on code generation tasks, up from baselines as low as 22.5%.
  • Substantial Quantitative Gains: CoCoEvolve yielded up to a 37.91% improvement on non-overlapping test sets for chart-to-code reproduction and up to 46.88% on complex out-of-domain chart modifications.
  • Reduced Computational Overhead: By using test-time consistency signals to select the best candidate rollouts, CoCoEvolve actually reduced API evaluation costs substantially (e.g., dropping from \$0.8637 to \$0.2382 per sample compared to sequential baseline generation).

Limitations and Future Work

  • Model Scale Constraints: The framework was primarily validated on smaller-scale model architectures (ranging from 1.3B to 8B parameters). The authors note that exploring how co-evolution dynamics scale with massive models (e.g., 480B+) remains an open question.
  • Static Images Only: Currently, CoCoEvolve only operates on static chart images. Extending the framework to handle dynamic or animated visualizations—where temporal semantics across frames introduce extra complexity—is highlighted as an exciting future direction.

Summary updated

Abstract

As chart images, tabular data, and visualization code play increasingly important roles across diverse domains, cross-representation understanding across these modalities poses fundamental challenges for AI systems: the relationships across representations are inherently \textit{one-to-many}, supervision is ambiguous and costly, and model optimization lacks a principled signal that is both direction-adaptive and representation-generalizable beyond task-specific objectives. We introduce CoCoEvolve to improve consistency across chart, table, and code representations. Instead of treating cross-representation mapping as a one-to-many problem, we define explicit one-to-one correspondences and optimize models using agreement between representations, without additional annotations. During training, CoCoEvolve@Train performs co-evolution across the chart-table-code cycle, while CoCoEvolve@Test applies the same consistency objective at inference time for test-time co-optimization. We also present CoCoEvolve@Eval, an evaluation suite covering all six cross-representation tasks. Across four benchmarks, CoCoEvolve improves performance in both training-time and test-time settings. Our project page: https://xhguo7.github.io/CoCoEvolve/.
BibTeX
@misc{guo2026consistency,
  author = {Xuehang Guo and Pengyuan Li and Tom Hope and Tirthankar Ghosal and Manling Li and Qingyun Wang},
  title = {{Consistency-Driven Co-Evolution for Self-Supervised Cross-Representation Learning}},
  year = {2026},
  eprint = {2608.04926},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2608.04926},
}
Figure