When a textbook says "the diagram above" or "this section," the reader has to find something that is not in any shared room. It is either a visible object on the page or an idea the text has built up earlier. In 1977, researchers of reading comprehension proposed that written language forces this shift. Speech lets speakers designate, meaning point at a thing both people can see. Writing forces readers to construct, meaning build the referent in their heads from the words alone. That distinction has never been tested with data. A newer line of work named the document-internal version of the phenomenon communicative deixis: pointing phrases such as "this chapter" or "these sentences" whose targets are parts of the document itself. That work catalogued such phrases in textbooks but stopped before the step that matters for any reading system, which is working out what each phrase actually points to.
This proposal builds the first benchmark for that resolution step. Each pointing phrase is linked to a target: a bounding box on a rendered page, a section node, or a span of earlier text. We then use the benchmark to turn the designation/construction distinction into a prediction that can fail. Pointing phrases that behave like designation should be solved mostly by page geometry. Phrases that behave like construction should be solved mostly by reading the preceding text, and should gain little from layout. If the benchmark shows that crossover pattern, the old theory gains its first quantitative support, and we learn which signal a document reader actually needs for each kind of reference. If it does not, the distinction is not doing computational work.
Core Claim
We hypothesize that the four-way taxonomy of Determiner-Established Deixis to Communicative Artifacts in Pedagogical Text has three resolvable types, and that they split along the designation/construction axis. That paper classifies pointing phrases in 122 Wikibooks as structural, illustrative, discourse, or non-communicative, with expert agreement of Cohen's kappa 0.70.
Illustrative references ("that diagram," "the table above") are designation on a page. The target is a visible object, and 2D position should carry most of the information. Structural references ("this section") point into the document's outline tree. Discourse references ("this argument") are construction in the strict sense of From Conversations to Stories: An Analysis of Differences Between Oral and Written Language Comprehension. The referent is assembled from earlier clauses, much like the intra-textual deixis that Some Facts about Centers, Indexicals, and Demonstratives documented for "that" in spoken interviews.
The testable claim is an interaction. Removing layout information should hurt illustrative resolution much more than discourse resolution. Removing the text context before the phrase should do the opposite.
We also operationalize the "oral language bug" hypothesis of D. D. McDonald and BruceLanguage Generation: The Linguistics Component. Those papers stated it as theory with no data: learners fail when they carry conversational heuristics into print. The computational analogue is a system trained on dialogue deixis, where recency dominates. End-to-End Neural Discourse Deixis Resolution in Dialogue reports that over 90% of antecedents in dialogue fall within four utterances. We predict that such a system will fail on a specific slice: illustrative references whose target is far away in reading order but close on the rendered page, such as a floated figure placed above the paragraph but serialized many paragraphs later.
Research Program
The data come from the same Wikibooks source Wilson and Oberlander used. Their counts fix the scale. About 8% of the corpus's more than 114,000 sentences contain candidate "determiner + noun" phrases, and about 48% of those were judged true communicative deixis. That implies roughly 4,000 real instances. We re-render the printable HTML of these books into page images and keep each page's element geometry and outline tree. Annotators then link a stratified sample of about 3,000 instances to their targets, with each type represented. A target is one of three things: a figure or table box, an outline node, or an antecedent text span, which may be a clause or a multi-sentence stretch. Doubly annotated subsets give agreement figures for resolution, alongside the existing classification agreement.
To test generalization beyond pedagogical prose, we add a smaller annotated set drawn from the PIN-Arxiv and PIN-PMC subsets of PIN: A Knowledge-Intensive Dataset for Paired and Interleaved Multimodal Documents (m-a-p/PIN-200M), which pairs Markdown with page images.
Because Wikibooks is HTML, we can re-render the same text under different layouts: floats above or below, one or two columns, and figures moved across page breaks. The words stay identical and only the geometry changes. This lets us perturb designation cues without touching construction cues. For training data we also create silver examples by rewriting explicit numbered references ("Figure 3") into determiner forms ("this figure"). The numbered label gives the gold target for free. We treat this only as augmentation, because the rewrite shifts the distribution, and all evaluation uses naturally occurring phrases.
The resolver is intentionally simple. It is a candidate-ranking model that scores each possible target, where candidates are page elements, outline nodes, and preceding clauses. The scorer combines three feature channels whose contributions we can switch on or off one at a time. The text channel is the phrase and its surrounding text. The layout channel is 2D offsets between the phrase and each candidate, plus page-break indicators, in the spirit of the spatial edges used by GRAVL-BERT: Graphical Visual-Linguistic Representations for Multimodal Coreference Resolution. The structure channel is the outline-tree path. The model is a measuring instrument for the interaction test, not an end in itself.
Baselines
The naive reference points are two proximity heuristics. The first picks the nearest candidate of the right type in reading order. The second picks the nearest candidate in 2D page distance, respecting "above" and "below" when the phrase says so. These run on identical candidate sets. We expect them to be strong on illustrative references and we report them per type. Beating them overall means little. What counts is beating them on the far-in-text, near-on-page slice and on discourse references.
The strongest existing text-only method is dd-utt from End-to-End Neural Discourse Deixis Resolution in Dialogue. It reports a macro-average CoNLL score of 48.5 on the CODI-CRAC dialogue benchmark, against a prior 38.3. We run it two ways: as published, trained on ARRAU plus dialogue, and retrained on our serialized document text with the same training data, tuning budget, and encoder size as our resolver. We also include the multi-task ensemble from Anaphora Resolution in Dialogue: System Description (CODI-CRAC 2022 Shared Task), which reports discourse deixis CoNLL F1 of 35.91% to 62.79% across tracks, as a second text-only competitor. Neither has a reported result on document communicative deixis. We found no reported result on Lacuna for communicative deixis resolution by any system.
For detecting pointing phrases before resolving them, the baseline is the most-frequent-sense heuristic. Wilson and Oberlander showed it misses 58% of true communicative senses, so we report detection separately and also evaluate resolution with gold phrases to keep the two error sources apart.
We add a general vision-language model given the page image and asked to box the referent, with the same page context our resolver sees. We found no reported result on Lacuna for such models on this task, but they are the obvious practical competitor.
The key ablations are our resolver with the layout channel removed, with the text channel limited to the phrase itself, and with the structure channel removed. We run each ablation separately for each reference type.
Why not just…?
Why not just use nearest-figure-in-layout? For illustrative references this is a real threat and might match any learned model. If it does, that is itself evidence for the designation reading, and we will report it as such. It cannot handle discourse references, which have no page box. It is also brittle under our layout perturbations whenever "above" competes with section boundaries. The per-type results settle how far it goes.
Why not just run a text coreference model on serialized text? DocVQA: A Dataset for VQA on Document Images shows that flattening pages costs a great deal on layout-dependent questions: 0.665 ANLS for BERT-large against 0.981 for humans. Dialogue deixis models are also built around recency, which we predict misfires on floated artifacts. The retrained dd-utt run tests this directly. If it matches the full resolver on illustrative references, the layout channel is not needed.
Why not just prompt a large vision-language model? It may well do well on illustrative references. It is still an uncontrolled instrument for the scientific question, because we cannot cleanly remove its access to layout or to text. We therefore include it as a baseline and test whether its errors follow the same layout-perturbation pattern as our ablations, but we do not use it to measure the interaction.
Proposed Experiments and Decisive Checks
The primary metric is top-1 resolution accuracy per reference type, where a prediction counts if its box overlaps the gold box by more than half or it names the correct outline node. For discourse targets we use span-level F1. The decisive test is a two-way comparison, channel ablated against reference type, with bootstrap confidence intervals over books.
The hypothesis is supported if removing layout drops illustrative accuracy by significantly more than it drops discourse F1, and removing preceding text shows the reverse. It is refuted if both types degrade by similar amounts under both ablations.
The second check uses the layout perturbations. For the same text, we move each figure relative to its pointing phrase. A designation-driven resolver's predictions should follow the figure, while its discourse predictions should not change. We run a small human study on the same perturbed pages, recording resolution accuracy and response time, to see whether human readers show the same asymmetry. This would be the first behavioral data bearing on the 1977 distinction.
The third check is the oral-bug slice. We compare the published dialogue-trained dd-utt against the retrained resolver on the far-in-text, near-on-page subset, and predict a gap concentrated there rather than spread across all references. Finally, we repeat the main comparison on the PIN arXiv and PubMed Central subset to see whether the crossover survives in research papers, where references are more often numbered and floats move more.
References
- Determiner-Established Deixis to Communicative Artifacts in Pedagogical Text
- From Conversations to Stories: An Analysis of Differences Between Oral and Written Language Comprehension
- Some Facts about Centers, Indexicals, and Demonstratives
- From Conversations to Stories: An Analysis of Differences Between Oral and Written Language Comprehension
- BruceLanguage Generation: The Linguistics Component
- End-to-End Neural Discourse Deixis Resolution in Dialogue
- PIN: A Knowledge-Intensive Dataset for Paired and Interleaved Multimodal Documents
- GRAVL-BERT: Graphical Visual-Linguistic Representations for Multimodal Coreference Resolution
- Anaphora Resolution in Dialogue: System Description (CODI-CRAC 2022 Shared Task)
- DocVQA: A Dataset for VQA on Document Images