Leave-Region-Out Stacking for Reading Place Names on Historical Maps

Historical maps are full of place names that a computer reads badly. The names are printed small, cross mountain shading and river lines, and are often spelled according to another language's rules: Spanish names in the American Southwest, French names in Louisiana, Dutch names in the Hudson Valley, Gaelic names in Scotland. These names have "deep" spelling, meaning the letters do not predict the sound in any simple way. When the recognizer misreads a letter or two, the usual fix is fuzzy spelling match against a gazetteer (a database of known places), and that fix breaks exactly where the spelling rules change. It also breaks when a system trained on maps from one region is run on maps from another. We propose a stacked ensemble for this problem. Several different readers each propose which gazetteer entry a map label refers to: a visual recognizer that keeps fine image detail, a spelling-distance matcher, and a pronunciation-based matcher. A small second-stage model, the meta-learner, learns which reader to trust for each label.

The new ingredient is how the meta-learner is trained. Standard stacking trains it on out-of-fold predictions, meaning each base model's predictions on data it did not see during training. The folds are usually random. On maps, random folds mix labels from the same sheet and region into training and test. That teaches the meta-learner how the base models behave on familiar ground, while deployment happens on unfamiliar ground. We instead build the folds by holding out whole regions. The meta-learner then sees how each reader degrades under geographic shift, and in particular how the spelling-based reader fails on foreign-origin names while the pronunciation-based reader holds up. The central claim is testable: leave-region-out stacking should beat both random-fold stacking and any single reader on unseen regions, and the gain should concentrate on names whose spelling conventions differ from the map's language.

Core Claim and What Is New

Three bodies of work each supply part of this idea, and none puts the parts together. The first is map-reading pipelines. mapKurator detects text on historical USGS, Ordnance Survey and NYPL maps, groups words into phrases, and links them to a knowledge base. Adding a visual U-Net predictor to a text-only grouping model raised word-linking F1 from 54.26 to 70.54. It relies on off-the-shelf recognition and does not model spelling depth or arbitrate among competing readers. SynthMap raises map text detection F1 on the David Rumsey collection from 57.32% to 64.25% using style-transferred synthetic maps, but it addresses detection only. These papers motivate the application and supply the detection stage we reuse.

The second is toponym resolution and name pronunciation. GeoNorm resolves place names in news text. It generates candidates by exact match, edit distance and character n-grams, then reranks them with BERT and population features, reaching 80.7% accuracy on the LGL news benchmark. Its candidates come from spelling surfaces, and it never sees images. Bhargava and Kondrak show that re-ranking the n-best outputs of three different grapheme-to-phoneme systems (systems that predict pronunciation from spelling) with an SVM that uses transliteration evidence cuts relative error by 12 to 27% on proper names. They also expose the constraint we design around: a re-ranker cannot recover an answer missing from the base systems' candidate lists. PanPhon supplies the method for pronunciation similarity. It maps IPA segments to 21 articulatory features, and adding those features raised zero-shot Uzbek-to-Turkish NER F1 from 2.09 to 11.9, which is evidence that sound-level representations bridge spelling divergence. The resulting design has two parts. Pronunciation-based matching is a candidate generator that works independently of the recognizer's beam, which relieves the n-best bottleneck. Its output is combined with the other readers by stacking, not by a pipeline cascade.

The third is geographic shift and stacking. Saha, Banerjee and Zhu show two things on satellite change detection. Diverse training cities beat nearby ones (Kappa 0.32 vs 0.12). A scalar logit-confidence switch can hand control to a fallback model on unfamiliar scenes (Dubai: 0.06 supervised, 0.18 fallback). The switch is hand-thresholded and only relative. Desloires et al. and Osgood-Zimmerman et al. use stacking over tabular geospatial features to handle out-of-year and subnational variation. Neither builds folds by region to target spatial shift, and neither involves text. Mission Critical and the Sentinel-1 flood deployment paper document why random splits overstate accuracy under spatial autocorrelation. That is our argument for spatial folds applied at the stacking level, not only at evaluation.

Research Program

We will build a benchmark and a system. For the benchmark, we will annotate transcriptions and gazetteer links (GeoNames plus period gazetteers where available) for map labels from the David Rumsey, USGS, NYPL and Ordnance Survey collections that the cited papers already use. The maps will be grouped into about five regions chosen for distinct etymological layers: the US Southwest (Spanish), Louisiana and the Great Lakes (French), the Hudson Valley (Dutch), Scotland and Wales (Gaelic and Welsh), and a control region of mostly English-origin names. Each label also gets an origin tag from a character-level origin classifier, spot-checked by hand, so results can be broken down by spelling depth.

The system has three stages. The first detects text with a SynthMap-style detector. The second runs three base readers, each producing a scored candidate list of gazetteer entries. The visual reader is a recognizer on a high-resolution backbone, using an HRFPN-style pyramid or DDCM-style dilated blocks that avoid downsampling, whose strings are matched to the gazetteer. The layout gains that motivate this choice come from HRFPN on Tibetan manuscripts, which reached 72.7 AP against 24.7 for Mask R-CNN, and from DDCM-Net. The spelling reader is GeoNorm-style edit-distance and n-gram candidate generation over the recognizer's top-k strings. The phonetic reader converts both the recognized string and gazetteer names to IPA under several source-language grapheme-to-phoneme hypotheses (English, Spanish, French, Dutch, Gaelic rules through Epitran-style converters) and retrieves candidates by PanPhon feature-weighted alignment distance. The third stage is a gradient-boosted meta-learner over the union of candidates. Its features are each reader's scores and ranks, the recognizer's character confidences, the name-origin posterior, and map-level style statistics such as font size and background texture energy. We deliberately exclude raw coordinates, because they are meaningless in a held-out region.

The meta-learner's training set is built from leave-region-out predictions. Each base reader that has trained parameters is retrained once per held-out region, and only its predictions on that region feed the meta-learner. The fold-cost problem with high-resolution backbones is real: K regions means K extra backbone trainings. We will measure a cheaper variant that freezes a shared detector and backbone trained on all non-test regions and refits only the recognition head per fold. We will report whether the leakage this introduces measurably inflates meta-learner trust.

Baselines

The naive reference is off-the-shelf recognition followed by exact or lowercase gazetteer lookup, essentially the recognition-and-linking front half of mapKurator. It shows how much of the problem is solved without any effort.

The strongest existing competitor is a single pooled system. It is a recognizer fine-tuned on all training regions together, which follows the diversity finding of Saha et al., followed by GeoNorm-style candidate generation and a BERT reranker (GeoNorm), with map-context features in place of document context. It gets the same detector, training data, and total GPU budget as our system, and the same hyperparameter search budget, tuned on the same held-out validation region. GeoNorm reports 80.7% accuracy on LGL news text, and mapKurator reports linking F1 of 70.54 on USGS maps. Both numbers are for different tasks. We found no reported result on Lacuna for transcription-plus-gazetteer-linking accuracy on historical map labels under a region hold-out, so we will establish this baseline ourselves. Our system beats it if it improves held-out-region linking accuracy by at least three absolute points, averaged over region folds with bootstrap confidence intervals excluding zero, and does not lose on the English-origin control region.

Three further baselines isolate ingredients. Random-fold stacking uses the identical meta-learner and features but random, sheet-mixed folds. This is the ablation for the key claim. Score averaging and max-confidence switching across the three readers is the Saha-style alternative to a learned meta-learner. Stacking without the phonetic reader isolates the pronunciation contribution. A backbone swap replaces the high-resolution recognizer with a standard stride-32 backbone at matched parameters, which tests whether preserving resolution matters for small, diacritic-bearing labels.

Why not just…?

Why not just train one strong reader on diverse regions? This is the most serious threat. Saha et al.'s Kappa 0.32 vs 0.12 result says diversity alone buys much robustness, and a large pooled model may absorb the Spanish or French spelling patterns implicitly. The pooled baseline above tests this directly. If it matches our system on held-out regions, and especially on foreign-origin names, then the stacking is unnecessary and we will say so. We expect it to fall short mainly when the held-out region's spelling layer is absent from training, which is the fold where Dutch or Gaelic is withheld. That is the case the phonetic reader's language-parametrized rules are meant to cover.

Why not just average the readers or switch on confidence? Averaging assumes each reader is equally reliable everywhere. The Saha switch uses a single relative score that its own authors note is not calibrated across scenes. Bhargava and Kondrak's learned re-ranker clearly outperformed raw base scores on names. The averaging and switching baselines will show whether that advantage survives on maps.

Why not just use edit distance, as GeoNorm does? Edit distance treats "Guadalupe" misread as "Gwadaloupe" as far from the right answer, while pronunciation-aware distance places it close. PanPhon's cross-lingual NER result is indirect evidence that this matters. The phonetic-reader ablation, broken down by name origin, settles the question. If the phonetic reader adds nothing on foreign-origin names, the pronunciation component should be dropped.

Why not just use random folds, which is how stacking is usually done? Random folds are cheaper. If they perform as well as region folds, the geographic framing adds nothing. That comparison is the paper's central experiment.

Proposed Experiments and Decisive Checks

We will run the full leave-one-region-out protocol and report linking accuracy, character error rate of the chosen transcription, and oracle candidate recall, meaning whether the correct entry appears in any reader's list. Four results would decide the claim. First, region-fold stacking must beat random-fold stacking on held-out regions. We also predict that random-fold stacking will look better on in-region validation, which would reproduce the evaluation leakage the geospatial literature warns about, now at the meta-learner level. Second, the phonetic reader must raise oracle recall over recognizer-derived candidates, directly addressing the n-best bottleneck. Third, gains must concentrate on names whose origin differs from the map language and be near zero on the English control region. Fourth, the high-resolution backbone must lower character error on labels below a fixed pixel height relative to the stride-32 swap. We will also report the cost of the frozen-backbone fold shortcut and the meta-learner's calibration on held-out regions, since a stacked model that is confidently wrong in new territory would repeat the failure it was built to fix.