Borrowing Variability: Transferring Intra-Writer Differences Instead of Generating Signatures

When a signature verifier sees only one genuine sample of a new person, it does not know how much that person's signature normally varies. The current fix is to generate synthetic copies of that sample. DeepWriteSYN does this with a stroke-level generative model and cuts one-shot error roughly in half (DeepWriteSYN). Diffusion-based handwriting synthesis gets strong recognition results but needed 62 hours on 8 GPUs to build its corpus (GC-DDPM). We propose a much cheaper alternative. Other writers in the training set have many genuine samples, so their variability has already been observed. We store the differences between pairs of genuine signatures from those writers as vectors in a learned feature space. A one-shot reference is then expanded by adding borrowed difference vectors to its embedding. Every source writer with $N$ genuine samples contributes $N(N-1)/2$ such differences. This is the same quadratic pair counting that makes dissimilarity representations data-efficient (Dissimilarity Representation for Handwritten Signature Verification). The difference is that the pairs are mined from other people and spent on the person who has none. No pixel or pen trajectory is ever generated.

The question that makes this worth doing is a scientific one: is intra-writer variability transferable across people and across writing systems? Cross-script tests in signature verification are where models break. Competing architectures collapse to 29–50% equal error rate when trained on Hindi and tested on English (DetailSemNet). No generative augmentation method in the file has been tested across datasets at all. If a bank of differences built from Latin-script writers helps a one-shot Bengali writer, then intra-class variability has a shared, script-independent structure that a verifier can reuse cheaply. If the bank does not help, we learn that variability is script-bound. That would also suggest generative synthesis must be retrained per script, which is exactly where its cost hurts.

Core Claim

Our hypothesis is that structured, real intra-writer differences can stand in for generative synthesis in one-shot signature verification. Composed onto a one-shot reference embedding, they should recover most of the error reduction that generative synthesis gives, at a cost several orders of magnitude lower. They should also degrade more gracefully than generated samples when the bank comes from a different dataset or script.

What is new is not pairwise training itself. Siamese verifiers such as SigNet already train on pairs. The new step is reusing the observed genuine-pair differences at enrollment time to build a per-writer acceptance region. That region is built in feature space and drawn from a bank whose source can be controlled: same dataset, different dataset, or different script.

Research Program

We first train a writer-independent encoder $f$ on a source dataset with ordinary pairwise contrastive training. For every source writer $w$ with genuine samples $x_{w,1},\dots,x_{w,N_w}$, we store the genuine difference vectors $d = f(x_{w,i}) - f(x_{w,j})$ for all $i \neq j$. We store forgery differences $f(\text{forgery}) - f(\text{genuine})$ the same way. For a new writer with a single reference $r$, we form pseudo-genuine embeddings $f(r) + \alpha d$ and pseudo-forgery embeddings $f(r) + \alpha d'$. Here $d$ is drawn from the genuine bank, $d'$ from the forgery bank, and $\alpha$ is a scale tuned on validation writers. A small per-writer scorer is then fit on these points. Following Pairwise Difference Learning, the scorer predicts "same writer" from concatenated features $[z_q, z_a, |z_q - z_a|]$ against the composed anchors. This also gives an entropy-based uncertainty estimate across anchors.

The design choice we expect to matter most is which differences to borrow. Writers differ in how much they vary. A careful signer and a hurried one should not share a variability budget. We therefore compare three variants. The first samples from the whole bank. The second samples only from the $k$ source writers whose embeddings are nearest to the reference. The third projects differences onto a writer-shared subspace, in the spirit of separating shared from private traits (Discriminative Factorized Subspaces). The anchor-selection problem flagged for pairwise methods becomes concrete here: we will measure how many borrowed differences are needed before error stops falling.

Cost accounting is part of the program, not an afterthought. For every method we report total wall-clock and GPU-hours for three stages: training any generator or bank, per-writer enrollment, and per-query scoring.

Baselines

The do-nothing reference is the writer-independent encoder alone. It scores a query by distance to the single reference, with no enrollment-time expansion. This is the SigNet-style pairwise setup, retrained by us on identical splits. Every other method must beat it, and the gap over it is the effect we are claiming.

The strongest generative competitor is DeepWriteSYN, which synthesizes 12 variants of a single online signature. On DeepSignDB (BiosecurID subset), it reduced one-shot equal error rate from 19.79% to 9.36%, as reported in DeepWriteSYN. We will run it with 12 and with more synthetic samples, through the same encoder and scorer as our method, on the same writers. If we cannot obtain the code, we compare against the reported numbers only under their protocol and state this caveat. Beating it means equal or lower EER at matched enrollment count, with generation cost reported alongside. Matching it at a small fraction of the cost would already be a positive result.

On the offline cross-script side, the strongest reported zero-shot result is DetailSemNet trained on BHSig-H and tested on CEDAR, at 7.32% EER (DetailSemNet). That result uses real enrollment samples rather than one-shot enrollment, so it is a ceiling to approach rather than a like-for-like competitor. We will also apply our bank on top of a DetailSemNet-style encoder to test whether the gain stacks with a stronger backbone.

Two ablations isolate the key ingredient. The first replaces borrowed differences with isotropic Gaussian noise matched in norm, which tests whether the structure of real variability matters or only its spread. The second replaces the bank with differences from shuffled writer labels, so that pairs are cross-writer rather than genuine. All methods share the encoder, the train/test writer splits, and an equal hyperparameter-search budget on held-out validation writers.

Why not just…?

Why not just rely on the writer-independent Siamese model, which already learns from pairs? This is the real threat. A well-trained encoder may already map genuine variability into a tight cluster, leaving nothing for enrollment-time expansion to add. The evidence against complacency is that SigNet falls to 77.76% accuracy on GPDS Synthetic (SigNet), and baseline distance matchers collapse cross-script (DetailSemNet). Both failures are consistent with a single fixed threshold ignoring per-writer variability. Our do-nothing baseline settles this directly. If the bank gives under one EER point over it in every setting, we will report the idea as negative.

Why not just apply elastic image distortions, or a deformation-invariant distance? Neuromuscular variation is largely smooth warping, and Measuring dissimilarity with diffeomorphism invariance offers a closed-form distance invariant to such warps, with Nyström approximation keeping cost to $O(N^{1.5})$. It has never been tested on signatures. We include random elastic augmentation of the reference and this invariant distance as cheap competitors. If either matches the bank, then variability is mostly geometric and borrowing is unnecessary. That would be a clean finding in its own right.

Why not just use a classical kinematic synthesizer like Sigma LogNormal? It is cheap, but DeepWriteSYN reports an improvement of over 50% relative to it (DeepWriteSYN). We include it on the online track as the low-cost generative point, which places our method on a cost–error curve between it and DeepWriteSYN.

Proposed Experiments

The online track uses DeepSignDB with one-shot enrollment. It compares the bank, DeepWriteSYN, Sigma LogNormal, and the ablations by EER, together with GPU-hours per stage.

The offline track uses CEDAR, BHSig-B, BHSig-H, and GPDS. Here we cross bank source and test target in all pairings: same dataset, different dataset of the same script, and different script. For example, a bank built from GPDS is applied to BHSig-H writers with one reference each.

The decisive check is a 3×3 comparison of bank origin: same-script, other-script, and shuffled-label banks. If other-script banks recover most of the same-script gain while shuffled banks recover none, intra-writer variability transfers across writing systems. In that case, cheap borrowing is a credible replacement for per-script generation. If other-script banks behave like shuffled ones, variability is script-specific, and the cost of generative synthesis is harder to avoid than the field hopes.

As a secondary check, we test robustness to document noise on Tobacco-800. There, Offline Signature Verification on Real-World Documents reports 0.17 EER. This shows whether borrowed differences confuse stamp and scan artifacts with genuine variation.