Compilers as Eyes: Using Strict-Language Checks to Fix Diagram Perception in Multimodal Code Models

Models that write code from software diagrams, such as UML class diagrams (the boxes-and-arrows drawings that list each class, its fields, its methods, and whether each member is public + or private -), do much better in Python, JavaScript and PHP than in Java, C# or Scala. The usual reading is that this is ordinary programming-language bias: there is less training data for strict languages. We propose a different explanation that can be tested. Much of the gap may come from misreading the picture, and permissive languages hide those misreadings. A Python program that drops a private marker or loosens a method signature still passes its unit tests. A C# program with the same mistake fails to compile, or is caught by a check of its declared members. On this view the scripting-language advantage is partly an artifact of a lenient grader, and the strict compilers that seem to cause the gap are actually the cheapest and most detailed source of feedback about whether the model saw the diagram correctly.

The proposal turns that observation into a training signal. We generate diagrams from known ground-truth specifications, have the model write code in strict languages, and score the output with a structural check: compile it, then inspect its classes, members, visibilities and signatures, and compare them against the specification. This gives a dense reward for each member. We use it to update only the part of a vision-language code model that turns image features into language-model inputs, plus the LayerNorm scales inside attention. If the explanation is right, a visual front end trained this way should improve every output language, including Python, when judged by a strict oracle. It should also leave the code model's text-only ability intact.

Core Claim

In diagram-to-code generation, two failures are mixed together. One is failing to read the picture (perception). The other is failing to express a correctly read design in the target language (realization). Execution-based benchmarks measure only their combination, and the combination is graded more leniently in dynamic languages. We claim three things. First, a large fraction of strict-language failures on diagram benchmarks are perception failures. Second, the compiler and a reflection-based structural check localize those failures to individual diagram elements. Third, rewarding the projector and normalization layers with that signal closes more of the gap than supervised fine-tuning, execution-based reranking, or a transcribe-then-generate pipeline.

The anchor observation comes from From Diagrams to Code, which trains M²-CODER on Qwen2-VL-7B using supervised learning only. That paper reports both the scripting-over-static bias and failures to pick up visibility symbols and abstract signatures. It treats the two as separate findings. We treat the second as a candidate cause of the first. Flow2Code reports similar language sensitivity across 15 languages but is an evaluation and supervised fine-tuning study. PPOCoder shows that compiler and test rewards raise CodeT5's compilation rate from 52.14% to 97.68%, but it works on text only and updates the whole policy. We borrow its reward idea and move it to a different target: we reward the visual interface, and the reward checks fidelity to the diagram rather than just whether the code compiles.

Research Program

Diagnosis. M²C-INSTRUCT diagrams were rendered from Mermaid and PlantUML source (Multilingual Multimodal Software Developer), and Flow2Code flowcharts were rendered from DOT and Visustin. For both, the true structural specification is therefore known. We build three probes for each item.

  • Perception probe. The model writes a language-neutral schema of classes, members, visibilities, types and relations. We score it against the source specification.
  • Realization probe. The same model receives the ground-truth specification as text, with no image, and writes code in each target language.
  • Strict-oracle audit. We re-score passing M²EVAL solutions in Python, JavaScript and PHP with a structural checker. The checker verifies declared members and naming or visibility conventions against the diagram. It is necessarily weaker for dynamic languages, which do not enforce these rules, and we will report exactly what it checks.

The first two probes split each language's error into a perception share and a realization share. The audit measures how many lenient-language passes contradict the diagram.

Training signal. For a generated program $y$ in a strict language against specification $S$, we use one reward:

$$r(y,S) = \mathbb{1}[\text{compiles}] \cdot \Big(\alpha \cdot \tfrac{|M(y)\cap S|}{|S|} + (1-\alpha)\cdot \text{pass}(y)\Big)$$

Here $M(y)$ is the set of (class, member, visibility, signature) tuples read from the compiled artifact by reflection or by the compiler's symbol table, $|S|$ is the number of such tuples in the specification, $\text{pass}(y)$ is the unit-test pass fraction when tests exist, and $\alpha$ weights structure against behavior. The structural term is dense and cheap, and it applies to every synthetic diagram even when no unit tests exist. This addresses the sparse-reward problem that dominates execution-based reinforcement learning.

What gets updated. We optimize this reward with a PPO-style objective and a KL penalty, as in PPOCoder. Only two groups of parameters change. The first is the vision-to-language projector. The second is the attention LayerNorm parameters, following Tuning LayerNorm in Attention, which shows that adapting only these can outperform LoRA by more than 20%. Keeping the rest frozen tests whether the gains come from perception rather than from rewriting the code model. It also guards against the forgetting documented in Deep Pre-Alignment for VLMs, where text reasoning fell from 84.8 to 36.4 on MATH-500 under standard multimodal training. We train on Java, C# and Scala and evaluate on all ten M²EVAL languages. Improvement in held-out lenient languages is the key evidence that we corrected perception rather than taught language-specific generation.

Resolution ablation. Token pooling such as the 2×2 merge in MiniGPT-v2 could erase the small +/- glyphs. We therefore run the full pipeline both with and without pooling, to see whether our gains simply reproduce what higher resolution gives.

Baselines

Reference points. The naive reference is the base Qwen2-VL-7B under prompting. The strongest open supervised baseline is M²-CODER-7B, which From Diagrams to Code reports at 25.3% average Pass@1 on M²EVAL across ten languages. That is above Qwen2-VL-72B at 24.7%, and well above 18.0% for the version trained without the first alignment stage. GPT-4o at 49.7% and Gemini-2.5-Flash at 48.7% are reported ceilings, not training baselines. The file gives no per-language M²EVAL Pass@1 numbers, so we will re-run M²-CODER ourselves to get per-language breakdowns.

Equal-compute supervised fine-tuning. We continue supervised fine-tuning of M²-CODER on exactly the synthetic strict-language diagrams used for our reward training, with the same trainable parameters and the same GPU-hours. This is the direct competitor. It uses the same data and has the correct answers available, so any gain we show must come from the reward rather than the data. We need gains in both Pass@1 and strict-oracle Pass@1 on held-out languages.

Full-policy execution reward. A PPOCoder-style baseline applies the same reward to all parameters, or to LoRA adapters, instead of our restricted set. This tells us whether restricting updates to the visual interface matters.

Behavior-only reward. This ablation sets $\alpha=0$, keeping compile and test rewards but dropping the structural term. It isolates the key ingredient.

Transcribe-then-generate pipeline. The model reads the diagram into PlantUML text, and a strong text code model (Qwen2.5-Coder) writes the code from it.

Inference-time reranking. We sample $n$ candidates and pick one by execution consensus, in the style of SRank and VRank, with $n$ matched to the rollout compute our method spends.

For every method, beating a baseline means a higher mean Pass@1 on M²EVAL and Flow2Code-UML over three seeds, with non-overlapping bootstrap intervals. We also require a smaller strict-minus-dynamic gap when both are judged by the strict oracle. Finally, text-only Pass@1 on MBXP and Multilingual HumanEval from Multi-lingual Evaluation of Code Generation Models must not drop by more than one point.

Why not just…?

Why not just transcribe the diagram to PlantUML and use a text code model? This is the most serious threat. The 0% that text-only models score on M²EVAL reflects deliberately incomplete text prompts, not a failed transcription, so that result does not rule the pipeline out. Our realization probe measures this pipeline's upper bound directly. If perfect transcription recovers most of the strict-language gap, the bottleneck is transcription, and the useful contribution becomes compiler-rewarded transcription rather than end-to-end generation. We would report it as such. Flow2Code's roughly 10.6% drop on pseudocode flowcharts suggests that intermediate text abstractions add their own errors, but that is suggestive evidence, not decisive.

Why not just rebalance training toward strict languages? Unimax balancing in Measuring the Impact of Programming Language Distribution helps low-resource text generation, with a +66.48% pass@k gain, but it cost high-resource languages 39.70% at 1B scale. More importantly, rebalancing addresses realization, not perception. The equal-compute supervised baseline on strict-language data is exactly this alternative. If it matches our method on held-out lenient languages, the perception explanation is wrong.

Why not just rerank samples with execution at inference time? Reranking can only pick among what the model already produces. If the model reliably misreads a glyph, every sample carries the same error, and consensus will confirm it rather than correct it. The compute-matched reranking baseline tests this claim, and we will report whether reranking closes the strict-oracle gap.

Why not just raise image resolution? This may help. The pooling ablation measures it, and higher resolution combined with our reward is a legitimate outcome if both contribute.

Decisive Checks

We ask four questions, each settled by a specific outcome.

  1. Do lenient passes hide misreadings? The strict-oracle audit must show that a substantial share of Python, JavaScript and PHP passes on M²EVAL violate the diagram's structure. If they rarely do, the premise fails and we fall back to studying realization.
  2. Is strict-language error mostly perception? The perception-versus-realization split must attribute most strict-language errors to perception. Otherwise, repairing the visual interface is the wrong lever.
  3. Does the fix transfer? Training on only Java, C# and Scala must raise strict-oracle Pass@1 in Python and the other held-out languages beyond the equal-compute supervised baseline. That transfer is the signature of fixing perception rather than language-specific generation.
  4. Is the structural reward the active ingredient? The behavior-only ablation with $\alpha=0$ must underperform the full reward. The model must also hold text-only MBXP performance within a point, which is evidence that the code model itself was left intact.