Synchronizing Only the Geometry: Low-Bandwidth Training of Curved-Text Spotters

Models that read text in photographs have to handle words printed on bottles, tilted signs, and circular logos. The strongest of these models contain small "geometry heads": layers that predict where a word bends, which way it is rotated, or how to unwarp it before reading. Training such models on millions of synthetic images across many GPUs means constantly exchanging gradients. On ordinary Ethernet or between data centers, that exchange takes most of the training time. The standard remedy is to let each worker train on its own for many steps and only occasionally average the copies' parameters. We suspect that remedy fails for text readers in a specific place. If two copies have drifted apart and you average their parameters, the averaged geometry head can predict a warp, a rotation angle, or a centerline that neither copy would produce. That error should hit curved and rotated text much harder than flat text.

The proposal is simple. The bulky shared feature extractor synchronizes rarely, in the style of DiLoCo. The geometry heads hold only a small fraction of the parameters, so they are averaged every step at almost no bandwidth cost. We also test a second hypothesis: a geometry head shared across all workers acts as an anchor that keeps the workers' feature extractors in compatible representations, so their periodic average stays usable. If both hold, a colleague could train a curved-text spotter over a 1 Gbit/s link and keep the accuracy of fully synchronous training on curved benchmarks. If neither holds, we will have the first measurement of whether infrequent synchronization damages spatial prediction at all.

Core Claim

The claim is specific. Under periodic parameter averaging, accuracy losses in text spotters concentrate in the geometry-predicting modules and in geometrically hard test subsets. Averaging only those modules every step recovers most of the gap to fully synchronous training, and the gain comes from what the modules do, not from how many parameters get synchronized. The communication cost is easy to state. Let $|\theta_b|$ be the parameter count of the backbone and recognizer, $|\theta_g|$ the parameter count of the geometry heads, and $H$ the number of local steps between backbone synchronizations. Bytes sent per worker per step are then roughly

$$ B \approx \frac{|\theta_b|}{H} + |\theta_g|, $$

up to the constant factor of the all-reduce. Because $|\theta_g| \ll |\theta_b|$ in a spotter built on ResNet-50, $B$ stays within a small factor of plain DiLoCo's $|\theta_b|/H$.

Grounding in Prior Work

Two bodies of work motivate this, and one sets limits on it. On the systems side, Scaling Laws for DiLoCo has replicas take $H=30$ local steps and then merges their parameter differences with an outer Nesterov optimizer. It reports compute utilization above 80% with 10–100× less bandwidth. WASH prevents replica drift by randomly shuffling weights between workers. Both were developed on transformers trained on text or images with a single objective. Neither looks at modules that regress coordinates, angles, or warping grids, where parameter-space averaging has no reason to produce a valid geometric output. Our proposal reuses DiLoCo's outer loop unchanged and replaces WASH's random weight exchange with synchronization chosen by module role.

On the Utility of Gradient Compression is the main constraint. It found that gradient compression gave a speedup in only 6 of more than 200 configurations, and that PowerSGD encoding alone costs about 45 ms per step on ResNet-50. The reason is that synchronous training already overlaps communication with the backward pass on fast networks. We take this result at face value: on InfiniBand-class clusters, plain synchronous training is the right answer for spotters of this size. Our regime is therefore links of 10 Gbit/s or less. The same paper notes that 4× compression already suffices for ResNet-101 at 10 Gbit/s, so the advantage should only appear at or below that bandwidth. Our method also avoids compression compute entirely, which sidesteps the GPU contention that paper identified.

On the text side, the method relies on explicit geometry outputs. FOTS regresses rotated boxes and unwarps feature maps with its differentiable RoIRotate operator. PGNet predicts maps of each word's centerline, its border offsets, and its reading-direction offsets, then gathers characters along the predicted centerline. Portmanteauing Features is evidence that this kind of geometry is fragile: a rectifier's bad control points can make readable text unreadable, which is why it adds an unrectified feature stream as a fallback. AON found it needed gates that weight its four reading-direction streams so that one orientation does not dominate. That suggests different replicas could settle on different orientation conventions. Mask TextSpotter v3 shows how much rotated text separates methods: its F-measure on RoIC13 improved by 21.9%. None of these papers studies distributed training.

The closest connecting work is MERIT. It splits conflicting instruction data across GPU clusters that never communicate, then merges their weights, and it is evaluated on benchmarks that include TextVQA. MERIT partitions data and merges once at the end. We partition parameters by function and synchronize continuously. We also study dedicated spatial models rather than a general vision-language model.

Research Program

The work proceeds in three stages. The first stage is diagnostic. We train each spotter with plain DiLoCo for $H \in \{10, 30, 100\}$. Just before each outer averaging step, we measure how far the replicas disagree functionally on a held-out batch: the spread of their predicted polygons, rotation angles, and centerline offsets. We then compare each replica's geometry error with the geometry error of the averaged model. If the averaged model is markedly worse than every individual replica on geometry outputs but not on character classification, that confirms the mechanism. We also check whether rotation-angle heads fail by wrapping around at ±π.

The second stage builds the hybrid scheme. The modules synchronized every step are FOTS's box and angle regressor; PGNet's border-offset and direction-offset heads and its graph refinement module; and, for recognizers, the control-point predictor of a thin-plate-spline rectifier, which bends the image onto a grid. Everything else synchronizes every $H$ steps through DiLoCo's outer optimizer.

The third stage tests the anchoring hypothesis. We measure how similar the replicas' backbone features are, using centered kernel alignment, as a function of local step count, with and without the shared heads. If the shared heads keep the backbones closer, the method has a second benefit beyond protecting the heads.

Training follows the usual recipe. We pretrain on SynthText for the spotters and on the 12M synthetic words for the recognizers, then fine-tune on the real benchmarks. Pretraining is where communication dominates. We use 8–16 GPUs grouped into islands, with inter-island bandwidth throttled to 1, 3, and 10 Gbit/s, and we report real wall-clock time.

Baselines

The naive reference point is synchronous data-parallel training with communication overlapped against the backward pass. It is run at each throttled bandwidth with the same data, number of steps, and learning-rate schedule. It sets the accuracy ceiling and the speed floor. For the strongest task models we take accuracy targets from the papers: an end-to-end F-score of 63.1% on Total-Text for PGNet, and 89.6% on CUTE80 and 86.4% on SVTP-None for SAR. Our reproductions under synchronous training are the operative reference. We report them next to the published numbers rather than assuming we match them.

The strongest communication-efficient competitor is plain DiLoCo from Scaling Laws for DiLoCo, with every parameter synchronized every $H$ steps. We give it the same tuning budget as our method over outer learning rate, outer momentum, and $H$. Its reported result is compute utilization above 80% at 10–100× lower bandwidth, measured on language models. No reported result was found on Lacuna for DiLoCo on scene-text spotting. A fair comparison matches total bytes sent, so DiLoCo is also run at a slightly smaller $H$ chosen to equal our method's byte budget $B$.

We also run DeMo with $k$ tuned to the same byte budget. It reports 85× less data sent per step on OLMo with no loss in accuracy, but it has never been tested on vision models. WASH is run with a shuffle fraction tuned to the same byte budget.

The key ablation synchronizes a random set of parameters of the same size as the geometry heads every step. A second ablation synchronizes only the character-recognition head instead. If the random-subset control recovers as much accuracy as the geometry heads do, the role-based claim is false.

Our method beats DiLoCo at an equal byte budget if two things hold. First, it closes at least half of DiLoCo's accuracy gap to synchronous training on Total-Text's curved subset, CUTE80, and RoIC13. Second, its gain over the random-subset control is larger than the variance across random seeds.

Why not just…?

Why not just use synchronous training, since Agarwal et al. showed it usually wins? On fast datacenter networks it does win, and we treat that as settled (On the Utility of Gradient Compression). The claim here is limited to links of 10 Gbit/s or less. Our wall-clock curves will show at which bandwidth, if any, synchronous training stops dominating for spotters of ResNet-50 size. If the crossover falls below any realistic bandwidth, the proposal reduces to the accuracy diagnosis alone, and we will report it that way.

Why not just run DiLoCo with a smaller $H$? This is the most serious alternative. A smaller $H$ shrinks drift everywhere, at a cost in bytes. The byte-matched DiLoCo run tests exactly this. If uniform synchronization at equal bandwidth matches our method, role-based synchronization adds nothing, and we will say so.

Why not just compress gradients with DeMo or low-rank projection? Compression leaves synchronization at every step, so it keeps the encoding overhead Agarwal et al. measured. DeMo discards gradient components outside its top-$k$ frequency coefficients, and fine stroke detail is what distinguishes curved characters from each other. The byte-matched DeMo run, broken down by subset, settles whether that loss happens in practice.

Why not just rectify offline and train a flat-text reader? Errors in offline rectification cannot be undone later (Portmanteauing Features). Joint training also lets the recognition loss help the detector learn to ignore background clutter (FOTS). This alternative also removes the geometry heads entirely rather than training them efficiently.

Decisive Checks

We evaluate on Total-Text end-to-end F-score, separately for curved and straight words; on ICDAR 2015 and RoIC13 F-measure; on recognition accuracy for CUTE80, SVTP, and IIIT5K; and on direct geometry error, measured as polygon IoU and angle error. We also report wall-clock time to reach the target accuracy and total bytes sent. Results are averaged over three seeds.

The proposal stands or falls on three results. First, does plain DiLoCo lose more accuracy on curved and rotated subsets than on straight ones? Second, does the averaged model show geometry error above that of every individual replica? Third, does synchronizing the geometry heads beat both byte-matched DiLoCo and the size-matched random subset? If the first result is negative, infrequent synchronization is safe for text spotters as it stands. That would be a useful finding, and the method would be unnecessary.