Margin-Triggered Index Updates for Revised Scientific Papers

Search systems over scientific literature, such as the vector databases behind retrieval-augmented question answering, are built from frozen snapshots of corpora like peS2o and S2ORC. Papers keep changing after they are indexed. They receive correction notices, move from preprint to published version, and get new captions, funding lines, or fixed numbers. Most of these changes are deliberately small. Correction notices are administrative patches that leave conclusions intact. The question is not whether to propagate them but how much work each one deserves. We propose to treat every indexed vector as a stale copy that is allowed to drift until the drift could actually change what a user retrieves, and only then pay for synchronizing the distributed index. This is the logic of event-triggered communication from control theory: a node stays silent until its true state departs from its last broadcast by more than a threshold. Here it is applied to index maintenance, with the threshold derived from retrieval score margins rather than chosen by hand.

The opportunity matters because the two obvious policies are both poor fits for sharded indexes. Rebuilding on a schedule re-does hours of work for edits that touch a grant number. Pushing every edit immediately into a sharded graph index triggers graph repair, segment compaction, and cross-worker coordination for changes too small to alter any search result. That coordination is exactly the kind of overhead that dominates small distributed operations.

Core Claim

For unit-normalized embeddings and cosine scoring, a stale vector $\hat{x}_i$ for document $i$ differs from its true current vector $x_i$ in score against any unit query $q$ by at most the drift norm, $|q\cdot x_i - q\cdot \hat{x}_i| \le \|x_i - \hat{x}_i\|$ (Cauchy–Schwarz). The top-$k$ membership of document $i$ for query $q$ therefore cannot flip as long as the drift stays below half its margin to the $k$-th score. We define a per-document trigger threshold over a reference query set $Q$:

$$\varepsilon_i = \tfrac{1}{2}\min_{q\in Q}\,|q\cdot \hat{x}_i - \tau_k(q)|,$$

where $\tau_k(q)$ is the $k$-th highest score for $q$ in the current index. A revision is committed to the distributed index only when the cumulative drift $\|x_i - \hat{x}_i\|$, measured against the last committed vector, exceeds $\varepsilon_i$. The factor of one half covers the worst case in which a competing document drifts by a similar amount in the opposite direction.

Our central empirical claim is about the workload. Because correction notices are scope-bounded, most real revisions produce drift far below their margin. A margin-aware trigger can then skip the large majority of index synchronization events while keeping top-$k$ results on reference queries identical to a freshly rebuilt index, and close to identical on held-out queries.

Two things are new. The first is the threshold, which depends on each document's position relative to retrieval boundaries rather than on a uniform tolerance. The second is a measured account of how much index work scope-bounded revisions actually require.

Relation to Prior Work

The mechanism comes from event-triggered coordination. In consensus protocols, an agent broadcasts only when $\|x(t)-\hat{x}(t)\|$ exceeds a threshold (Distributed event-triggered coordination for average consensus on weight-balanced digraphs). Emulation-based estimators keep a shadow copy of what the receiver believes and cut traffic by up to 78% (Event‐based state estimation: an emulation‐based approach). Event-triggered diffusions make the threshold state-dependent (Synchronization in Networks of Linearly Coupled Dynamical Systems via Event-Triggered Diffusions). We reuse the shadow-state idea directly: the indexed vector is the shadow. Our state-dependent threshold is a retrieval margin instead of a Lyapunov quantity. None of these works addresses index maintenance or the cost of hardware synchronization.

The closest systems work is Manu: A Cloud Native Vector Database Management System. Manu replaces strict transactions with a log and a user-chosen bounded-staleness window driven by periodic time ticks. It bounds staleness in time, whereas we bound it in retrieval effect and send nothing when nothing meaningful changed. Delayed-synchronization traversal (Accelerating Graph-based Vector Search via Delayed-Synchronization Traversal) and pipelined training with one-step gradient delay (FLSGD: free local SGD with parallel synchronization) show that relaxing barriers pays off. Both relax synchronization during computation, not during index maintenance, and neither uses an error bound to decide when to resynchronize.

The overhead constraint comes from Exploring Distributed Vector Databases Performance on HPC Platforms: A Study with Qdrant and its companion dataset paper (Biological Query and Pes2o Corpus Embedding Dataset). On peS2o with up to 32 workers on Polaris, these papers find three things. Distributed querying is slower than a single worker below about 30 GB. Scaling from 1 to 4 workers on a shared node gives only 1.27x. Inference accounts for 98.5% of embedding runtime. These findings motivate the design: small, frequent distributed operations are where overhead hurts, so suppressing unnecessary ones should pay.

The workload assumption, that revisions are mostly bounded, rests on the correction-notice literature. Most notices fix affiliations, names, authorship roles, or grant numbers. The scientific fixes that do occur are localized, such as a replaced figure panel (Correction: An ion-gating synaptic memristor based on tri-layer HfOx composition regulation) or a numerical typo whose authors state the conclusions are unchanged. The FLSGD erratum is a fitting test case. Correction to: FLSGD: free local SGD with parallel synchronization adds grant numbers and rewrites one caption to state that the cluster has "16 workers, and their performance is not identical." The first change should never touch an embedding. The second might, slightly.

Research Program

We will build an update path in front of a sharded Qdrant deployment. When a revision arrives, the system re-embeds only the changed chunks; this is a single-document cost and cheap. It then computes cumulative drift against the committed vector and compares it with the stored $\varepsilon_i$. Sub-threshold revisions update only a metadata sidecar and the drift ledger. Revisions above threshold enter a commit queue that is merged into the shards in batches. The batch size is chosen to stay above the small-operation regime in which coordination dominates.

Margins are recomputed lazily, and only for documents near a commit. This keeps the per-document thresholds cheap. Repeated tiny revisions to one paper cannot cause a burst of commits, which is the index analogue of Zeno behavior, because drift is always measured against the last committed vector rather than the previous revision. A minimum interval between merges handles the remaining edge cases.

Building the revision workload is part of the contribution. We will hand-apply the stated changes from a stratified sample of real correction notices to their parent papers. The strata follow the notice categories in the file: affiliation, name, authorship role, funding, license, figure panel or caption, and numerical typo. We will then fit a synthetic edit generator to the observed category frequencies and edit sizes so that the workload scales to corpus level. As a harder, more substantive stress set, we will use preprint versus version-of-record pairs drawn from the multi-version records described in S2ORC: The Semantic Scholar Open Research Corpus.

Baselines

The naive reference point is doing nothing: the index is never updated after the snapshot. This measures how much retrieval actually degrades from revisions. If it barely degrades, the whole question is moot, and we will report that.

The first strong competitor is immediate upsert: every revision is re-embedded and written into the sharded index as it arrives. This is what a careful engineer would build today. Manu shows such systems keep low latency at 3k–4k insertions per second, so per-update throughput is not the bottleneck. We are betting on the cumulative cost of graph repair, compaction, and cross-shard coordination. We will run it on the same Qdrant cluster configuration, the same revision stream, and the same embedding model.

The second competitor is periodic rebuild, which is also the time-tick policy in the spirit of Manu. Revisions accumulate, and the index is refreshed every $T$ hours, with $T$ tuned over the same grid budget we give our threshold. For the full-rebuild extreme, the file reports insertion of the roughly 80 GB peS2o set in about 21.67 minutes with 32 workers, and a 21.32x index-construction speedup from 1 to 32 workers (Qdrant study). For incremental scholarly-index maintenance specifically, we found no reported result on Lacuna.

The third competitor is a scope rule. A classifier routes administrative corrections to the sidecar and always re-embeds and commits content corrections. Two ablations isolate our ingredients. The first replaces $\varepsilon_i$ with one global threshold tuned on validation data, which tests whether the margin-awareness matters. The second removes the sidecar, which tests whether drift measurement alone suffices.

Beating these baselines means matching immediate upsert's recall@10 relative to a freshly rebuilt oracle index, within bootstrap confidence intervals, on held-out queries. It also means using substantially fewer index commits and cluster CPU-hours than immediate upsert, and beating periodic rebuild on recall at equal compute.

Why not just…?

Why not just upsert every revision immediately? Correction notices are rare, so this may simply be cheap enough, and that is the main threat to the proposal. Our case rests on the observation that small distributed operations are where coordination overhead dominates (Qdrant study, sub-30 GB inversion). Graph indexes also degrade under repeated deletes and reinserts. We will measure total cluster CPU-hours, merge stalls, and query latency during updates at realistic correction rates and at stress rates that include preprint-to-published revisions. If immediate upsert is within noise at realistic rates, we will say so and narrow the claim to high-revision regimes.

Why not just ignore administrative corrections and re-embed the rest? The scope rule probably captures most of the savings, because most notices are administrative. It fails in two directions. It always commits localized scientific fixes, such as caption rewrites and typo fixes, even when they move nothing. And it has no safety guarantee when a mixed revision slips past the classifier. The scope-rule baseline and the uniform-threshold ablation settle this question. If the scope rule matches us on both commits and recall, the remaining contribution is the measurement study and the bound, and we will report it that way.

Why not just rebuild periodically? Rebuild cost is dominated by embedding inference (98.5% of runtime in the Qdrant study), and most of that work is spent on documents that did not change. Periodic refresh also leaves meaningful revisions stale for up to $T$. The recall-versus-compute curve over $T$ against our threshold sweep will show whether any $T$ dominates.

Proposed Experiments

All experiments use peS2o embedded with Qwen3-Embedding-4B, served by Qdrant on a Polaris-like cluster of up to 32 workers across 8 nodes, matching the published configuration. Queries come from the 22,723 BV-BRC genomic terms, split into a reference set used for margins and a disjoint held-out set used for evaluation. The metrics are recall@10 and rank overlap against a freshly rebuilt oracle, index commits, cluster CPU-hours, and p99 query latency during updates.

The first decisive check is the drift distribution by correction category. We predict that funding, affiliation, and name corrections produce essentially zero drift once metadata is excluded from embedded text. We predict caption and typo fixes fall mostly below $\varepsilon_i$, and preprint-to-published pairs frequently exceed it. If caption and typo fixes routinely exceed margins, the bounded-scope premise fails for retrieval and the trigger buys little.

The second check tests the guarantee itself: on reference queries, zero top-$k$ flips should occur under the trigger. On held-out queries we will report how often flips occur, which measures how well the reference-set margins generalize.

The third check is the cost–recall frontier of all baselines and ablations, swept over revision rates from realistic correction volumes up to full preprint-revision streams.

This experiment is where the proposal stands or falls. If the trigger does not dominate both the scope rule and immediate upsert somewhere on that frontier, the right conclusion is that scholarly indexes should use the simpler policy, and the paper will say so.