Expert-written rules are cheap prior knowledge and expensive mistakes. Decades of knowledge-engineering work found that when specialists are asked to state their know-how as explicit rules, they leave out exceptions and over-generalize. The rules they produce are useful but noisy. Federated learning trains one model across many organizations without pooling their data. It is now being used to inject such rules into training at each participating site, and it usually treats each site's rules as ground truth. We propose the reverse direction. The federation itself becomes the instrument that checks the rules. Every site holds labeled data that silently confirms or contradicts each rule. If sites share only tiny per-rule tallies of how often a rule fired and how often it was wrong, a server can estimate how trustworthy each rule is, globally and per site. It can flag rules that are wrong at some sites but right at others, and propose narrower versions of broken rules. No raw records or model weights need to leave a site for this auditing step.
This matters most at the sites that need rules most. A small clinic or plant with fifty labeled cases cannot tell from its own data whether a guideline is reliable. Pooled tallies from thirty other sites can. The core claim is concrete. In data-scarce, non-identically distributed federations, rule weights estimated from pooled per-rule evidence beat two alternatives: rules trusted with a fixed hand-tuned weight, and rule weights tuned on each site's own data alone. The gain should grow as local data shrinks and as rule noise grows.
Where this sits relative to prior work
The motivating evidence comes from the knowledge-acquisition literature. A Pragmatic Knowledge Acquisition Methodology documents how little expert time is available for interviews. Noise and Knowledge Acquisition shows that forcing experts to verbalize rules introduces omissions and brittle edge cases. Our method borrows its refinement step from apprentice systems. LEAP checks individual steps against an incomplete theory and generalizes the valid ones. DISCIPLE-1 turns single examples into heuristics in domains without a complete theory. We keep their "revise the theory from concrete cases" idea, but the cases are scattered across private sites and only counts are revealed.
The closest work is Knowledge-Injected Federated Learning (FLwKM). Each client wraps the global model with local "knowledge models." Range-type rules forbid certain classes through an infinite logit mask, and a point-prediction model is mixed in with a manually tuned trust weight. FLwKM reports a "U-shape" in that weight and leaves adaptive tuning open. Its hard mask also cannot recover if a rule is wrong. Knowledge flows one way: rules shape the model, but nothing ever corrects the rules. We keep FLwKM's rule format and setting, and replace the fixed mask and fixed trust with weights estimated per rule and per site from federated evidence.
Two other federated works use rules differently. Federated Neuro-Symbolic Learning (FedNSL) induces logic rules statistically through federated variational EM over a fixed predicate set. It involves no human-authored rules to audit. Towards Rule-Based Knowledge Sharing in Federated Learning (RFL) builds rules on the server by discretizing uploaded class prototypes. In RFL, rules serve as a communication format between architecturally different models, not as expert knowledge whose correctness is in question.
Two centralized methods supply our training objective and constrain its design. Guiding Semi-Supervision with Constraint-Driven Learning (CODL) uses domain knowledge as soft penalties and reaches roughly 86% citation-segmentation accuracy with 20 labels. Posterior-regularized REINFORCE shows that rules covering about 4% of instances still give useful inductive bias. Both assume the rule weights are fixed and correct. That is exactly the assumption this proposal relaxes.
Core mechanism
A rule $r$ has a condition $c_r(x)$ and an allowed label set $S_r$, meaning "if $c_r(x)$ holds, the label is in $S_r$." This is FLwKM's range-type rule. Each round, site $m$ evaluates its rules on its labeled data and computes two numbers per rule. The first, $n_{m,r}$, is how many examples satisfy the condition. The second, $v_{m,r}$, is how many of those examples violate the rule. Sites send these counts through secure aggregation, optionally with differential-privacy noise.
The server fits a hierarchical Beta-Binomial model. Each rule has a global precision distribution, and each site has its own precision drawn from that distribution. This yields a shrunken per-site estimate $\hat\pi_{m,r}$ of rule $r$'s precision at site $m$. A site with few covered examples inherits the global estimate. A site with many covered examples, or with a markedly different violation rate, keeps its own.
Local training then uses a soft constraint weighted by these estimates:
$$ \mathcal{L}_m(\theta) = \mathcal{L}^{\text{CE}}_m(\theta) + \beta \sum_r w(\hat\pi_{m,r}) \, \mathbb{E}_{x \in \mathcal{D}_m : c_r(x)}\big[\, p_\theta(y \notin S_r \mid x) \,\big] $$
Here $\mathcal{L}^{\text{CE}}_m$ is the ordinary cross-entropy on site $m$'s labels, $p_\theta$ is the shared model's predicted class distribution, and $\beta$ is a single global strength. The weight $w(\hat\pi)=\log\big(\hat\pi/(1-\hat\pi)\big)_+$ is the log-odds of the rule being right, clipped at zero. Rules no better than chance therefore exert no pull. The constraint terms are also applied to unlabeled local data, which is where rules add information beyond the labels. The model weights themselves are aggregated by ordinary federated averaging, so the method is a layer on top of standard FL, not a replacement for it.
The refinement step closes the loop back to the experts. When a rule's global precision estimate falls below a threshold, the server proposes a small set of specializations. Each specialization adds one extra condition to the rule, such as a threshold on one feature drawn from a shared candidate grid. Sites return violation and coverage counts for each candidate, and the server keeps a specialization only if it restores high precision with acceptable coverage. This is top-down rule specialization in the spirit of LEAP and DISCIPLE-1. The output is a human-readable list of rules, each marked as confirmed, site-specific, demoted, or narrowed, which an expert can accept or reject. Communication per round is a few integers per rule and candidate, negligible next to model weights.
Research Program
We will build three things. The first is the counting and aggregation protocol, with secure aggregation and an optional Laplace or Gaussian mechanism on the counts. The second is the hierarchical rule-reliability estimator. The third is the weighted soft-constraint client and the specialization search.
The analysis has two parts. First, we derive when pooled evidence helps. For a rule covering $k$ examples per site across $M$ sites, the standard error of a pooled precision estimate scales as $1/\sqrt{Mk}$ instead of $1/\sqrt{k}$, which follows from binomial variance. This benefit only holds when sites agree, so we measure where the pooling bias from site-specific rules starts to dominate. Second, we quantify how much differential-privacy noise on the counts the estimator can tolerate before it stops distinguishing good rules from bad.
The key experimental device is controlled rule corruption. Ground-truth rule quality must be known, so we construct clean rule sets and corrupt them in three ways. "Broadened" rules have an over-wide condition. "Flipped" rules have a wrong allowed set. "Locally invalid" rules are correct at most sites and false at a chosen subset. As a more realistic condition, we also use rules written from feature descriptions alone, without access to data. We then measure how far rule noise, rather than label scarcity, explains the performance of each method.
Baselines
The naive reference point is plain federated averaging with no rules, trained on the same partitions, for the same number of rounds, with the same local optimizer and learning-rate sweep. Any rule method must beat it for rules to be worth using at all. FLwKM reports that standard FL violated expert rules in up to 12% of cases on its industrial coking data, so we also report the violation rate against the clean rules.
The strongest existing method is FLwKM Knowledge-Injected Federated Learning. It is the only federated method that injects expert rules of exactly our type, and it was evaluated on Covtype, which we reuse. Its reported Covtype result is a rise in client accuracy from 65% to 71% when knowledge models are injected, with the largest margins at 1% of training data. FedNSL and RFL cannot be run on hand-authored rules, so FLwKM is the right head-to-head competitor. We run it with the same corrupted rule sets as our method. Its trust weight gets the same tuning budget, a grid of ten values selected on pooled validation data. That selection is generous to FLwKM, since a real federation could not see pooled validation data. Beating FLwKM means higher mean and worst-site accuracy at the 1% and 5% data levels under nonzero rule noise, while staying within one point of it when rules are clean. On clean rules, FLwKM's hard mask should be close to ideal.
We also run CODL-style fixed soft constraints in the federation: one shared weight for all rules, tuned on the same budget. This isolates whether per-rule estimation matters beyond merely softening the mask.
Two ablations isolate the key ingredient. The first estimates rule reliability from each site's own counts only, with no pooling. Comparing it to the full method measures the value of federation itself. The second pools globally with no per-site component. It should fail on locally invalid rules, which tests whether the hierarchical model is needed. A third ablation disables refinement, to measure whether narrowed rules recover accuracy that demotion alone loses.
Why not just…?
Why not just tune the rule weight on each site's own validation data? This is the most serious alternative, and at data-rich sites it may well match our method. Our claim lives at the scarce end. With a few dozen covered examples, local estimates of a rule's precision are too noisy to separate a 95%-reliable rule from a 70%-reliable one. The local-only ablation, swept over local dataset sizes from 1% to 50% of Covtype, directly measures where the curves cross. If pooling shows no advantage even at 1%, the central claim is false, and we will report that.
Why not just use soft constraints with one fixed weight, as CODL does? Soft penalties already tolerate occasional violations, and CODL shows strong gains with them. However, CODL's own paper and the file's open-problems summary both note the same risk. A systematically wrong rule given nonzero weight keeps pushing a self-training loop in the wrong direction. Its pull is diluted, not removed. With a single weight, the federation must either underweight good rules or overweight bad ones. The fixed-soft baseline under the "flipped" corruption tests this, and we expect it to degrade roughly linearly in the fraction of corrupted rules while our weights zero those rules out.
Why not just drop the rules and use plain federated averaging? FLwKM's data-scarcity results argue that rules carry real information when labels are rare. If our weighting is near-optimal, the method should never fall meaningfully below the no-rules baseline even when most rules are corrupted. That "never worse" property is itself a decisive check.
Why not have experts validate rules directly, or have a language model write and check them? Expert time is exactly the scarce resource identified in A Pragmatic Knowledge Acquisition Methodology. Our output is a short, ranked list of suspicious rules, which spends that time where it matters instead of replacing it. Language-model checking is weak at precisely the numeric threshold comparisons these rules contain: RvLLM reports formal checking improving error detection by 15.7 to 50.2 percentage points over LLM-as-a-judge. We nonetheless include language-model-authored rules as one corruption source, because they are a realistic stand-in for unreliable experts.
Proposed Experiments
The primary benchmark is Covtype. We partition it across 30 sites with Dirichlet label skew and with geographic-feature skew. We build a clean rule set from shallow trees fit on a held-out slice that never enters training, and then apply each corruption type at 0%, 20%, and 40% of rules. The local labeled fraction is swept from 1% to 50%, with the remainder treated as unlabeled. The metrics are mean and worst-site accuracy, violation rate against the clean rules, and area under the curve for detecting corrupted rules from the estimated precisions. For specialization, we measure the fraction of broadened rules recovered to within the clean rule's precision.
As a second, structured-prediction setting, we federate the citation-segmentation task from Guiding Semi-Supervision with Constraint-Driven Learning, splitting by citation style so that some constraints are genuinely site-specific. The centralized reference is CODL's approximately 86% accuracy with 20 labels. Fashion-MNIST, which FLwKM also used, is a sanity check, although no reported numeric result was found on Lacuna for it. The industrial coking data would be ideal but appears proprietary, so we will request it and otherwise omit it.
Three outcomes would be decisive. First, at 1% local data with 20% or more corrupted rules, the full method should beat both FLwKM, even with its pooled-validation tuning, and the local-only ablation on worst-site accuracy. Second, on locally invalid rules, the hierarchical estimator should keep a rule active at the sites where it holds and deactivate it at the others, which global-only pooling cannot do. Third, corruption-detection AUC should remain above 0.9 under a differential-privacy budget of epsilon around 1 on the counts. If the first outcome fails, federated rule auditing offers nothing over local tuning. If the third fails, the privacy story is too weak for the regulated settings that motivate the work.
References
- A Pragmatic Knowledge Acquisition Methodology
- Noise and Knowledge Acquisition
- LEAP: A Learning Apprentice for VLSI Design
- DISCIPLE-1: Interactive Apprentice System in Weak Theory Fields
- Knowledge-Injected Federated Learning
- Federated Neuro-Symbolic Learning
- Towards Rule-Based Knowledge Sharing in Federated Learning
- Guiding Semi-Supervision with Constraint-Driven Learning
- Posterior-regularized REINFORCE for Instance Selection in Distant Supervision
- RvLLM: LLM Runtime Verification with Domain Knowledge