Checking Instructional Diagrams Against Their Own Geometry

Teachers increasingly have AI systems draft lecture slides and videos, and the cheapest reliable way to do this is to have a language model write drawing code rather than paint pixels. Code is cheap to run, easy to edit, and easy to inspect. Geometry and physics lessons, however, depend on diagrams being spatially right: the line called parallel must be parallel, the point said to lie on the circle must lie on it, and a "general" triangle must not happen to be isosceles. Today's pipelines either trust the generator or ask a vision-language model to look at the rendered picture. Vision-language models are known to be close to blind to spatial relations. This note proposes to skip the pixels. Run the drawing program, read exact coordinates out of the resulting scene, and turn them into a set of formal geometric facts. Then use those facts for three jobs: catching errors before a teacher sees the slide, indexing a bank of reusable diagram programs, and rewarding a compact geometry-specialized generator during training.

In two sentences: a diagram produced by code carries its own ground-truth geometry, so its spatial correctness can be checked exactly, the way unit tests check code, rather than judged by a model that cannot see "left" from "right." We build that checker and test whether it beats every judge-based alternative. We then use it as the retrieval key and the training reward for a 7B-scale diagram author working under teacher control.

Core Claim

Spatial verification for teacher-curated instructional content should run on the scene graph produced by executing the diagram program, not on the rendered image. The scene graph is the list of points, segments, circles and labels with their exact coordinates. A deterministic relation extractor over this graph checks three things. It confirms which requested relations hold. It identifies which claims in the accompanying narration are false. It flags which unintended special-case relations appeared by accident. We claim this extractor outperforms vision-language and language-model critics at catching spatial errors. We further claim that its predicates, used as the sparse half of a hybrid retriever and as a reinforcement-learning reward, let a compact specialized model produce diagrams that teachers accept without edits more often than a larger general model does. What is new is not any one ingredient but the use of a single exact geometric signal for checking, retrieval, and training inside an educator-in-the-loop authoring system.

Where This Sits

Two systems define the application setting. TeachMaster turns lecture outlines into videos by having agents write rendering code, fine-tuned on 3,735 annotated pairs for Qwen3-32B. It reports that 75.2% of pages needed no manual intervention. Its layout agent, though, only resolves clutter and overlap with heuristics, and it has no retrieval. LearnLens grounds feedback in a curriculum memory and checks outputs with a separate foundation-model verifier, but it is text-only. We keep TeachMaster's code-as-intermediate design and LearnLens's verify-then-revise loop. We replace the model-as-judge with an exact checker and add retrieval.

The motivation for refusing pixel-level judges comes from the spatial-reasoning literature. What's "up" with vision-language models? shows that zero-shot models score near chance on controlled preposition tests, with the best fine-tuned model at 56% against 99% for humans. It traces the problem to spatial words appearing in under 0.2% of web captions. Visual Spatial Reasoning finds a plateau around 70% against 95.4% for humans, and finds that models given explicit box coordinates do better. That second result directly motivates reading coordinates instead of pixels. Vision Language Models See What You Want but not What You See reports essentially no gain from scale on perspective-taking. Kamath et al. also constrain the design: fine-tuning vision-language models on spatial data tends to teach shortcuts. We therefore never train the checker. It is code.

Two lines of work supply the method. GeoX shows that a 7B model aligned to a formal geometry language (statements like Line A B C) beats GPT-4o on MathVista-GEO, 72.6% versus 66.1%. Its 6,232 expert-annotated formal diagram descriptions give us both a predicate vocabulary and ground truth. GeoX solves problems and does not author teaching material; we reuse its formal language as the target of our relation extractor. For training, unit-test-driven RL for regular expressions lifts LLaMA-13B from 0.41 to 0.53 pass rate by rewarding executed test outcomes. AutoTriton applies group-relative policy optimization to verifiable GPU kernels. Our geometric checker plays the role of their test harness. Specializing Smaller Language Models towards Multi-Step Reasoning warns of a specialization tax: its math-specialized Flan-T5 loses nearly all chain-of-thought ability outside math. We measure that tax explicitly.

For retrieval, Spatial-RAG is the closest bridge. It pairs structured spatial filtering with dense text scoring over map data and reports a 75.4% recall gain over embedding-only retrieval on proximity queries. It stops at geographic coordinates and relies on GPT-4-Turbo. We carry the idea from maps to diagrams, with geometric predicates as the sparse side. Mechanically, we follow Hybrid First-stage Retrieval Models for Biomedical Literature, which writes BM25 as a sparse vector and concatenates it with a dense one so a single nearest-neighbor search serves both. That paper also shows how badly dense-only retrieval handles exact technical tokens: 0.1173 MAP for the dense model against 0.2718 for BM25 on one BioASQ batch. Predicates like perpendicular(AB,CD) are exactly such tokens.

Research Program

The relation extractor. The first artifact is a deterministic relation extractor. Diagram programs are written in a restricted plotting dialect, Manim-like, matching TeachMaster's output. Executing a program records every primitive and label with coordinates. A tolerance-based routine then computes GeoX-vocabulary predicates over all object pairs and triples: collinearity, incidence, parallelism, perpendicularity, equal lengths and angles, tangency, and containment. It adds viewer-frame relations such as left-of and above for label placement, plus label-to-object attachment.

Tolerances matter. A relation that holds to within a pixel will look true to a student, so tolerances are set in rendered units. Their sensitivity is reported.

The three uses of the extracted facts. A teacher prompt such as "triangle with an altitude from A, not isosceles, angle at C obtuse" is parsed into requested predicates and forbidden ones. The narration is parsed into claimed predicates by the generator itself, with a separate audited parser as a check. The extracted predicate set $S$ is then compared against these. We define the training reward as

$$r = \mathbb{1}[\text{runs}]\left(\frac{|R \cap S|}{|R|} - \lambda \frac{|C \setminus S|}{\max(|C|,1)} - \mu\,|U \cap S|\right),$$

where $R$ is the set of requested predicates, $C$ the set of predicates claimed in narration, and $U$ a fixed list of salient special-case relations (unintended equal sides, right angles, parallels) not in $R$. The weights $\lambda$ and $\mu$ trade off narration accuracy and accidental special cases against coverage.

The same comparison runs at authoring time. Any slide where $r$ falls short of perfect is returned to the generator for revision, or surfaced to the teacher with the specific failed predicate named. This is LearnLens's revise loop with an exact judge.

The hybrid retriever. The second artifact is a diagram-program bank built from GeoQA, UniGeo and GeoX's annotated diagrams. Each stored program is indexed by two things. The first is its extracted predicate set, expressed as a BM25 sparse vector over predicate tokens with object names canonicalized. The second is a dense embedding of its descriptive text. Following Ma et al., the two are concatenated so one search returns exemplars that match both the teacher's intent and the requested geometry. Retrieved exemplar programs are placed in the generator's context.

The specialized author. The third artifact is a compact author: a 7B code model, using the LLEMMA-7B lineage as GeoX does. It is first supervised on bank programs paired with synthetic teacher prompts, then trained with group-relative policy optimization on the reward above. Retrieval stays on during training, so the model learns to adapt exemplars rather than draw from scratch.

Baselines

The naive reference point is TeachMaster-style generation with no retrieval and only an execution check. Here the program merely has to run, which is what TeachMaster's debugging agent enforces, and the generator is a prompted open model in the 32B class. This is roughly what a practitioner would deploy today. Our method beats it if, on held-out prompts, requested-relation recall rises and narration-claim error and unintended-relation counts fall, all with non-overlapping bootstrap intervals.

The strongest existing competitor for the checking task is a frontier model acting as critic. We run it two ways: shown the rendered image, and shown the program source. The image version is what LearnLens-style verification does in practice. The source version is the stronger and more dangerous competitor, because a text model can in principle compute slopes from coordinates. Both critics receive the same slides, the same teacher prompt and narration, and a tuned prompt with worked examples, and we give them the same number of prompt-iteration rounds we use to set our tolerances. Our checker wins if its F1 at detecting injected spatial errors exceeds both.

For the generation task, the strongest existing method is a frontier model given our retrieval and asked to self-critique from source. On the retrieval side, the strongest reference is the ME-HYBRID sparse-plus-dense scheme of Sparse, Dense, and Attentional Representations for Text Retrieval, which reports 0.343 MRR@10 on MS MARCO passages against 0.302 for a single dense encoder. That is text retrieval, and there is no reported result found on Lacuna for retrieval of diagram programs or for instructional diagram generation scored by geometric relations. We therefore re-run sparse-over-raw-text plus dense hybrid as the retrieval baseline on our bank, giving it the same embedding model and the same tuning of the fusion weight.

Ablations isolate the key ingredient. One removes the predicate sparse index while keeping dense retrieval. One replaces the geometric reward with execution-only reward at equal RL compute. One sets $\mu = 0$ to test whether the accidental-special-case penalty matters. One substitutes the image-critic score for the geometric reward, which tests whether exact verification, rather than RL itself, drives the gains.

Why not just…?

Why not just ask a frontier model to critique the drawing code? This is the real threat. Spatial blindness is documented for pixels (What's "up", VSR), not for coordinates written in text. A model reading Line((0,0),(4,1)) might well compute slopes correctly. Our critic experiment settles it. If the source-reading critic matches our checker's detection F1, the checking contribution shrinks to cost and determinism, and the proposal's weight shifts to retrieval and training. Our expectation that it falls short rests on chained computations. Programs often place points through transformations, intersections and loops, and recovering coordinates then requires executing the code mentally, which is where the regex results of Mao et al. show prompted large models stall at 0.43.

Why not just generate declarative constraints and let a solver place the points? This guarantees the requested relations by construction, much as GeoX hands formal programs to a symbolic solver. It does not guarantee the absence of accidental special cases, since a solver happily returns an isosceles triangle when the constraints allow it. It also does not check that the narration matches the picture, and it does not cover label placement. We include a constraint-solver authoring arm. If it closes most of the gap, the right conclusion is to keep the solver for layout and our checker for the unintended-relation and narration terms. That would be a narrower but still useful result.

Why not just use dense retrieval, or no retrieval with a few examples in the prompt? Dense encoders are weak on exact technical tokens: dense 0.1173 against BM25 0.2718 MAP in Ma et al.. Spatial-RAG shows large recall losses when geometric constraints are left to embeddings. Still, diagram banks are small, and fixed few-shot exemplars might suffice. The retrieval ablation measures this directly, through requested-relation recall of the top exemplar and through downstream generation quality.

Why not just fine-tune a larger general model on more diagram data? Supervised data teaches the look of correct programs, not whether a given program is correct. Execution-reward training beat larger prompted models on regexes (Mao et al.). We run a matched-compute supervised-only arm. If it equals the RL arm, the reward is not earning its keep.

Proposed Experiments

Benchmark construction. Because no benchmark for instructional diagram generation exists in the file, we build one from existing annotations. GeoX's 6,232 expert formal diagram descriptions and the GeoQA and UniGeo problem texts become teacher prompts with known requested predicates. Held-out splits are drawn by problem type, so retrieval cannot simply return the source diagram.

Error detection. We take correct bank programs and inject controlled perturbations: break a parallel, move a point off a circle, swap a label's side, make a triangle accidentally isosceles, or alter a narration sentence. The image critic, source critic and our checker are then scored on detection precision and recall.

Generation quality. Every arm is scored on requested-relation recall, narration-claim error, unintended-relation count and execution rate. As a solvability check, we feed each generated diagram with its problem text to GeoX and ask whether accuracy approaches GeoX's own reported 54.9% on GeoQA with original diagrams.

Specialization tax. We run the specialized author on non-geometry TeachMaster-style slides to measure what it loses there, as Fang et al. found it does outside the training domain.

Teacher study. Finally, mathematics teachers author short lessons with the full system and with the naive baseline. We record the fraction of pages accepted without edits, comparable in form to TeachMaster's 75.2%, and the time spent per page. We also record whether teachers find the named failed predicates actionable. Student learning gains, which LearnLens notes are rarely measured, are outside the scope of this study.

Decisive checks. Two outcomes decide the proposal. The checker must beat the source-reading critic on detection. The RL-trained 7B author must exceed the retrieval-equipped frontier generator on unintended-relation and narration-error rates while matching it on requested-relation recall. If the first fails, exact checking is not needed. If the second fails, specialization is not needed.