Reading the Block: Locating Storefront Photos by Matching Sign Order to the Street

A phone's GPS fix in a dense city is only good to roughly 10–40 meters. In that radius sit dozens of shops, offices and clinics, and several of them may be stacked on different floors of one building. Most systems try to decide "which venue is this photo of?" by reading the target's own sign and looking the name up. That fails exactly where it matters most. The sign may be occluded, stylized, or in a script the model reads poorly. The business may be one of three branches of the same chain within the radius. Or the photo may show a doorway with no name at all.

This proposal starts from a different observation. Even when the target's own sign is unreadable, the photo usually shows its neighbors. The left-to-right and bottom-to-top order of the signs in the frame is a physical fingerprint of one block face, the stretch of storefronts along one side of a street between two corners. A map database can supply the same fingerprint from above. We propose to read a street photo as a small graph of signs with their spatial relations, to pre-train an encoder on block-face orderings of points of interest (POIs, the map database's entries for shops and venues), and to identify the venue by aligning the two orderings. The alignment also infers which way the camera faced, since that determines whether "left in the image" means east or west.

Core Claim

Fine-grained venue identification within a GPS error radius should be treated as matching two orderings. One is the egocentric layout of signs extracted from the photo, meaning their arrangement as seen from the camera. The other is the allocentric arrangement of POIs along the surrounding block faces, meaning their arrangement in fixed map coordinates. It should not be treated as looking up one recognized string. Our specific, falsifiable prediction concerns the hard cases: an unreadable target sign, chain duplicates in the radius, and multi-floor buildings. In those cases, the order of neighboring signs supplies most of the disambiguating evidence. A model that keeps this order should therefore beat order-blind text matching by a wide margin on the hard cases and roughly tie it on easy ones.

A second claim concerns pre-training. GeoBERT pre-trains a masked model on POI categories inside geohash cells (square map tiles) that are flattened into sequences. It found that the choice of flattening order barely mattered. We read that as a sign that those orders were arbitrary, not that order is useless. A block face has a natural one-dimensional order, and it is the order a pedestrian's camera actually sees. Pre-training on that order should produce POI representations that align with photos far better than tile-based ones.

Research Program

The work has three parts.

Photo side. We build an extractor that reads street photos. Document-oriented visual information extraction already models the spatial relations between text blocks. OmniParser spots text and its polygon boundaries jointly with structure. GeoLayoutLM pre-trains on explicit direction, distance and collinearity relations between text blocks, which raised form relation-extraction F1 on FUNSD from 80.35% to 89.45%. Both were developed and tested on flat pages in normalized page coordinates. We reuse their method: a text-spotting front end, plus geometric relation features between sign instances. We retarget it to facades by grouping text instances into signs and assigning each sign a horizontal rank and a vertical band that serves as a floor proxy. We also keep logos and graphic marks as sign nodes even when they contain no readable text, since that kind of non-text artifact is precisely what document extractors drop. The output is a small sign graph whose nodes carry noisy text and appearance embeddings and whose edges carry relative direction and distance.

Map side. We pre-train a POI encoder in the domain. From a POI corpus with coordinates, names, categories and floor information where available, we construct block-face sequences by projecting POIs onto street segments and ordering them along the segment. The pre-training objectives are masked prediction of a POI's name and category from its block-face neighbors, plus prediction of relative floor for co-located POIs. This continues GeoBERT's self-supervised recipe with a physically meaningful order. It also follows the domain pre-training evidence from BERTHop, where in-domain language pre-training gave a 6.46% boost over general BERT. That paper motivates the choice but does not settle it for geography.

Alignment. We align the two sides with an explicit, discrete camera-orientation variable. Let $G$ be the sign graph from the photo, and let $c$ be a candidate (block face, position) inside the GPS radius with POI sequence $P_c$. We score

$$s(G, c) = \max_{o \in \{+1, -1\}} \; \max_{\pi \in \Pi_o} \sum_{i} \phi(g_i, p_{\pi(i)}) + \lambda\, \psi(G, \pi, P_c).$$

The symbols are as follows. The variable $o$ records whether the image's left-to-right axis runs with or against the block face's map direction. $\Pi_o$ is the set of order-preserving assignments of signs to POIs under that direction, with signs allowed to be unmatched. $\phi$ is a learned node similarity between sign $g_i$ and POI $p_{\pi(i)}$. $\psi$ rewards agreement between the image-space relations and the map-side spacing and floor relations, weighted by $\lambda$.

Making $o$ explicit targets the left-right inversions and symmetry traps that m2sv documents in vision-language models that compare street views to maps. Training is contrastive at the level of whole neighborhoods, not single image-text pairs. This responds to the evidence in Learning Robust Visual-Semantic Embeddings and Extending Multi-modal Contrastive Representations that pairwise pulling leaves the two modalities' overall distributions separate. The order-preserving assignment is solved by dynamic programming, so inference is cheap.

The closest prior system, BA-CNN, connects photos to reviews through mined visual concepts and uses GPS only as a hard filter. It reached 77.5% accuracy once GPS narrowed the choice to two candidates. It ignores sign layout and neighbors entirely. G3 grounds images in guidebook text, but only at country level and without reading text in the scene. Thinking with Map lets a vision-language model query map tools in text. The map's geometry never enters its representation. Our proposal differs from all three by making the spatial order of signs, matched against the map's order, the core evidence.

Baselines

The do-nothing reference is the GPS prior alone: pick the POI nearest the noisy fix, or pick uniformly within the radius. Its accuracy as a function of candidate count tells us how hard each stratum is.

The strongest simple competitor is order-blind text matching. We run the same text spotter, fuzzy-match every recognized string against the names of candidates in the radius with BM25 plus edit distance, and combine the result with the GPS prior. This is the baseline most likely to embarrass us on easy cases. It gets exactly the same spotter outputs, candidate sets and injected GPS noise as our method. Its fusion weights are tuned on the same validation split with the same number of trials.

The strongest learned prior method is BA-CNN's concept-plus-GPS design, reimplemented on our data. Its reported 77.5% (Yelp & Instagram, two GPS-narrowed candidates) is for a much easier candidate regime than ours, so we compare at matched candidate counts and do not borrow the number.

The strongest agentic system is a Thinking-with-Map-style map agent given the same map region. On MAPBench, ThinkingWithMap-Parallel reports 22.1% Acc@500m against Gemini-3-Pro's 8.0% in grounded mode. For storefront-level identification inside a 40 m radius, no reported result was found on Lacuna for any method, so our own benchmark has no external number to beat.

Three ablations isolate the key ingredients. Bag-of-neighbors keeps the neighbor set but discards order. Order-shuffled pre-training trains on randomly permuted block faces, mimicking GeoBERT's arbitrary flattenings. The fixed-orientation variant removes $o$.

Beating the field means two things. First, a clear accuracy gain over order-blind text matching on the hard strata (unreadable target sign, chain duplicates, multi-floor). Second, no loss on the easy stratum. A gain that appears only in aggregate would not count.

Why not just…?

Why not just OCR every sign and look the names up near the GPS fix? This is the real threat. When the target's sign is legible and unique in the radius, we expect it to match our method, and we will say so. The evidence against it as a full answer is indirect: flattening text into strings destroys relational structure (GeoLayoutLM), and Where on Earth? finds models failing on occluded or non-English text. The stratified evaluation settles the question directly. If order-blind matching ties us on the hard strata as well, the central claim is false.

Why not just give a strong vision-language model map tools? Thinking with Map shows that tools help a lot, but its absolute precision is low. Its 22.1% at a 500 m threshold is an order of magnitude coarser than a 40 m radius. Where on Earth? shows that search can also hurt: GPT-4o dropped 13.2% on country-level tasks when given it. We do not see the agent as a rival. Our matcher can serve as its final verification step, and we test exactly that plug-in on MAPBench.

Why not just fine-tune a general vision-language model end to end on photo–POI pairs? That would remove the need for specialized pre-training. m2sv is cautionary here. Supervised fine-tuning and reinforcement learning moved Qwen3-VL-8B only from 34.3% to 43.9% on viewing-direction identification, with persistent left-right inversions. We include this baseline with matched data and compute. If it closes the gap, the pre-training claim falls.

Why not just use GeoBERT-style tile pre-training? That is the order-shuffled ablation. It directly tests whether block-face order is what helps.

Proposed Experiments

No public benchmark in the file pairs street photos with GPS-radius candidate sets at storefront level. We will therefore build one. It will pair street-level imagery with POI records, inject GPS noise at 10, 20 and 40 m, and annotate each example for target-sign legibility, chain duplicates and floor. Cities are held out at test time, including cities with non-Latin scripts. This choice responds to the 42.7% Western-versus-Chinese performance gap in Where on Earth?. It also responds to the warning from EGO graph fusion that learned relational fusion degrades when the dataset changes. The primary metric is top-1 venue accuracy at each noise level, reported by stratum.

External checks come from three benchmarks. On MAPBench we report Acc@500m with the matcher plugged into an agent. On m2sv we use the orientation variable alone as a probe, judged against Qwen3-VL-8B's 43.9% and Gemini-3-Pro's 65.2%. On the GeoBERT urban tasks we check that block-face pre-training does not weaken POI representations of neighborhood function; these tasks report only relative gains, so the check is comparative.

Three decisive checks follow. First, if shuffling the sign order at test time does not hurt accuracy on the hard strata, the method is not using order. Second, if removing the orientation variable does not increase left-right swap errors, that variable is not doing its job. Third, if the pre-trained encoder's advantage vanishes in held-out cities, it has memorized local POI layouts and has not learned how block faces are structured.