Spending Endpoint Quota Where the Expert Needs It: Federated Evidence Retrieval for Live Fact Validation

Pipelines that extract facts from text or generate them with language models produce candidate knowledge-graph entries far faster than experts can check them. The natural way to check a candidate fact is to query the public linked-data sources that should either support it or contradict it. These are independent databases of subject–predicate–object triples, each exposing a SPARQL query endpoint. Those endpoints enforce fair-use quotas, typically a 60-second limit per query. Complete answers across a federation of endpoints can therefore take anywhere from seconds to tens of minutes. An expert reviewing a queue of candidates cannot wait that long for each item. In practice, either the expert validates without external evidence, or the evidence arrives too late to matter.

This proposal treats the expert's attention and the endpoints' quota as two budgets that must be scheduled jointly. The central observation is an asymmetry. Positive evidence found by cheap random-walk sampling over a federation is a real query answer, so finding it settles that the support exists. Showing that no contradicting evidence exists requires a complete query, which under quotas means slow chains of continuation queries. We propose a validation system with three rules. It uses sampling to find support and conflicts within a 3-second interaction window. It spends complete, quota-heavy queries only where completeness changes the decision. It sends to the expert only the items where federated sources genuinely disagree or evidence is missing.

Core Claim

A validation queue that routes each candidate fact by the type of evidence it needs can raise expert decisions per hour and cut endpoint requests compared with running a complete federated query per item. It should do so without increasing the rate of wrongly accepted facts. The routing is cheap sampled search for support and conflict, background complete search only to rule out contradictions or confirm absence, and human judgment for conflicts.

The novelty is not putting a human interface on top of a federated engine. It is a scheduling policy whose correctness rests on which conclusions sampled evidence can and cannot license.

Where this sits in prior work

Three systems motivate the design and supply its parts.

  • Complete answers under quotas. Continuation Queries: Embracing Timeouts on Public SPARQL Endpoints and Passage show that timed-out queries can be resumed statelessly. When a query hits its time limit, the server returns the partial results plus a standard SPARQL query encoding the remaining work. The cost is roughly 900 ms per checkpoint, and one Wikidata query took about 37 minutes over 37 continuations.
  • Source selection. FedUP uses a compact summary of the whole federation to discard 99% of useless endpoint combinations before sending any request.
  • Fast approximate answers. Fraw puts random-walk sampling on top of FedUP to answer autocompletion queries in under 3 seconds. It also acknowledges that an insufficient walk budget returns zero results.

All three serve a single query issued by a user who wants an answer. None asks which queries a downstream decision actually needs, and none treats an empty sample as anything other than an engine limitation. We reuse them as components and add the decision layer they lack.

Two systems connect validation to graphs.

  • Inferring Same-As Facts from Linked Data issues targeted federated queries to prove identity links between INA and DBpedia. Its human experts only audited 500 inferred facts after the fact. The humans never shaped which queries ran, and the method has no handling for quotas or timeouts.
  • CleanGraph provides an accept/reject/edit interface for graph corrections. It works only over a local MongoDB store and names RDF and SPARQL support as future work.

We place the human inside the live loop that CleanGraph built, but over the federated, quota-bound setting it cannot reach.

Two works on human effort constrain the design.

- Analysis of Automatic Annotation Suggestions finds that machine suggestions sped expert annotation by up to 35% without experts blindly accepting them. This supports showing pre-fetched evidence instead of a bare claim.

  • Data-IQ argues that human effort is best spent on genuinely ambiguous instances rather than easy or merely noisy ones. We borrow that lens: an item where two endpoints assert incompatible values is the federated analogue of an ambiguous instance.

Research Program

Evidence queries. For each candidate triple, the system compiles two families of SPARQL queries. Support queries look for the same assertion, or an equivalent one reached through identity links, at other sources. Contradiction queries look for incompatible assertions. Incompatibility is defined using a small, expert-declared set of constraints:

  • a single-valued predicate holding a different value;
  • a different-from fact for a proposed identity link, of the kind the INA–DBpedia work derives;
  • a type disjointness.

Both query families pass through FedUP source selection.

Fast stage. Within the interaction window, the system runs Fraw-style random walks for both families. Each item then lands in one of four states, which determine its route:

Sampling outcome Route
Support only Background stage, to confirm no contradiction exists
Conflict only Shown to the expert with a reject suggestion
Both Shown to the expert at once: a genuine disagreement that only judgment resolves
Neither Background stage, for complete search

Background stage. Contradiction and absence queries run to completion through Passage-style continuation queries, under a per-endpoint quota the system must respect. Unsupported operators such as negation are decomposed client-side, as in the Passage–Comunica extension. Two outcomes follow:

  • A support-only item whose complete contradiction search comes back clean is auto-accepted.
  • A neither item whose complete search still finds nothing goes to the expert flagged as unattested.

Scheduler. The scheduler decides which background continuations to advance next. Its priority combines two factors: how soon the expert will reach the item in the queue, and how likely completion is to change the item's route. It also reorders the expert's queue so that items whose evidence is ready come first.

We will build this as a federated backend for a CleanGraph-style interface. We will then measure where the time goes: expert waiting, expert deciding, background query time, and endpoints touched.

Baselines

Naive reference. The expert sees each candidate fact with no external evidence, the bare-claim condition of the annotation-suggestion study. It sets the floor for accuracy and the ceiling for raw speed.

On-demand complete evidence. This is the strongest existing approach, because it is what one would deploy today. For each item, as the expert opens it, the system runs complete support and contradiction queries through FedUP, with Passage continuations for any query that times out. The expert waits for the result.

The reference numbers we have are for query latency, not validation:

  • FedUP finishes the hardest FedShop query (q05) in 12.4 seconds, where CostFed needed over an hour.
  • Fraw reports that FedUP takes 23 seconds on autocompletion queries that Fraw answers in under 3 seconds.

We found no reported result on Lacuna for validation throughput or accuracy over a SPARQL federation, so that number must be established here.

Fairness conditions:

  • Both systems use the same endpoints, the same per-endpoint quota, and the same compiled evidence queries.
  • Both get the same lookahead prefetch over the next items in the queue, so the comparison isolates the routing policy rather than simple prefetching.

Beating this baseline means more correct decisions per expert-hour and fewer endpoint requests, with a false-accept rate no higher than the baseline's, within a pre-registered tolerance.

Ablations. Each removes one ingredient:

  • Symmetric sampling. An empty sample is treated as absence of contradiction, and the item is auto-accepted without the background check. This tests whether the asymmetry rule is what keeps false accepts down.
  • FIFO order. The scheduler is replaced by first-in-first-out ordering.
  • Model-confidence routing. Routing uses the extraction model's own confidence instead of federated evidence. This tests whether external evidence adds value beyond the generator's self-assessment.

Why not just…?

Why not just run all evidence queries in a batch overnight and validate afterwards? If the candidate set is fixed and a day's delay is acceptable, batch completion is a real threat. It removes waiting entirely. Its remaining cost is quota, because every contradiction query runs to completion even for items that sampling would have settled. We will report quota consumed by batch completion against our policy. We will also report results for a streaming setting where candidates arrive during the session. If batch matches us on quota in the static setting, we will say plainly that the contribution is confined to streaming queues and to quota savings.

Why not just use the sampled answers and skip complete queries? Because sampling cannot prove a negative, and Fraw itself notes that low budgets return nothing. Under that policy every unattested or uncontradicted item becomes a silent guess. The symmetric-sampling ablation measures exactly this. We expect it to inflate false accepts on facts whose contradicting evidence sits in selective joins. This is the same regime in which CRAWD reports that sampling estimates degrade.

Why not just let a language-model judge decide instead of an expert? The literature here says automated judges share correlated blind spots:

Our design already leaves the easy items to automation, and reserves the human for items where real endpoints disagree, which is where a judge has no privileged information. We will include a language-model judge on the conflict tier as an extra arm. That will show whether human review of those items is actually needed.

Why not just learn from execution which source combinations are empty, without any human? This is a fair point, and it is why we do not propose feeding human feedback into FedUP's summaries. Emptiness is observable automatically. The human is reserved for semantic conflicts, which no query can resolve.

Proposed Experiments

Synthetic setting with exact ground truth. On FedShop, with up to 200 vendor and review endpoints, we inject a controlled stream of candidate facts. Some are true facts withheld from one endpoint. Some are corrupted values. Some are values deliberately made to conflict across endpoints. Because the generator is known, accuracy can be measured exactly. Simulated experts (with response-time distributions and error rates calibrated from a pilot) allow sweeps over two variables:

  • per-endpoint quota, from 10 to 60 seconds;
  • the fraction of conflicting candidates.

Real setting. On LargeRDFBench's interlinked life-science sources (DrugBank, ChEBI, LinkedTCGA, a DBpedia subset), we perturb existing cross-source links to create candidates.

Reproducibility check against prior expert judgments. On the INA–DBpedia linkage task, we re-validate identity links produced by the import-by-query approach. Its 500-fact expert audit (100% precision) provides an external check on our auto-accept tier.

Metrics. - Correct decisions per expert-hour. - Expert idle time. - Time to first evidence on screen, with a target of 3 seconds. - False-accept rate of auto-accepted items. - Total endpoint requests and quota-seconds consumed.

Human study. A small study with real participants on FedShop, where no domain expertise is required, tests whether the timing model holds when actual people decide.

Decisive checks. The proposal succeeds only if three things hold:

  • the full policy beats on-demand complete evidence in decisions per expert-hour at equal or lower false accepts;
  • the symmetric-sampling ablation shows a measurably higher false-accept rate;
  • the full policy uses fewer quota-seconds than batch completion.

If the second check fails, the asymmetry rule is unnecessary. If the third fails, the idea reduces to prefetching.