An Append-Only Mixture-of-Experts Auditor for a Growing Smart-Contract Bug Taxonomy

Deployed smart contracts cannot be edited. A bug that ships stays exploitable, so the audit before deployment carries all the weight. The list of bug types auditors must catch also keeps growing. Early learned detectors targeted four classic bugs: reentrancy, timestamp dependence, integer overflow, and unsafe delegatecall. Later work showed that much of the real damage comes from other places. One is upgrade machinery, such as a proxy contract whose logic contract was never initialized and can be self-destructed. Another is concentrated admin power, such as a single key that can swap the contract's code with no timelock. Every time the bug list grows, a learned auditor must be retrained. A retrained model can quietly lose recall on bug types it used to catch, and in this domain a missed bug is permanent.

This note proposes building the auditor as a mixture-of-experts (MoE) model: a network whose feed-forward layers are split into many sub-networks ("experts"), with a small learned "router" choosing which experts process each token. The model is grown in an append-only way. A new bug family gets new experts and new router entries. The shared experts that encode general Solidity knowledge stay frozen, and so do the experts for earlier families. The claim is narrow and testable. Compared with retraining a dense model, this design should learn new bug families while keeping old-family recall and avoiding flipped verdicts on contracts already audited, at much lower cost per update. For serving, each release can be compressed back into one dense model with expert merging, while the full MoE is kept as the version that gets extended next time.

Core Claim

Two existing systems already use MoE for smart-contract security. SAEL gates three input streams: raw code, LLM explanations, and LLM verdicts. MOS turns an LLM's feed-forward layers into experts that specialize by vulnerability class. Both train once, on the same four classic bug types. Neither covers the upgrade and centralization defects documented by Li et al. and Lin et al.. Neither asks what happens when the taxonomy changes.

We do not propose "an MoE for upgrade bugs." Rule-based tools already handle much of that well, as discussed below. What we propose is to use MoE modularity for the auditor's own maintenance. When a new bug family arrives, the MoE structure lets us add capacity without overwriting the parameters that detect old families. A dense model cannot do this.

The design draws on two prior results. Li's attribution study of MoE models finds that always-on shared experts carry general knowledge and routed experts add domain-specific refinement. That finding motivates freezing the shared experts as the stable base. χFT shows that experts upcycled from a dense model can be merged back into a dense layer with no loss, and on HumanEval even a gain. That result supplies the serving step.

The two prior results also pull against each other. MOS argues that experts must stay separate. χFT argues that merging works better. We reconcile them by assigning each to a different job. Separate experts are kept for extending the model, and the merged model is what gets served.

Research Program

We start from a code LLM of about 7B parameters. It is continually pretrained on Solidity in the manner of MOS, then upcycled into an MoE with shared experts, following χFT. Stage 1 trains routed experts on the four classic classes, using the 6,524 explanations released with MOS and SAEL's corpus of more than 200,000 real-world contracts.

Each later stage introduces one new bug family. Stage 2 covers the six centralization-defect types from Lin et al., with silver labels from CDRipper on its 244,424 verified contracts and a hand-verified test subset. Stage 3 covers upgrade-mechanism defects: uninitialized logic contracts, upgrade functions that lack access checks, and storage-layout breaks along an upgrade chain. These are drawn from the 91,956 upgradeable contracts and 10,218 upgrade chains in Li et al. Stage 3 inputs are multi-contract: the proxy, the old implementation, and the new implementation are concatenated together.

At each stage, $m$ new routed experts are appended. The shared experts and all earlier routed experts stay frozen. The router grows by $m$ new rows. Old router rows are frozen, and a small replay set of earlier-stage contracts protects old routing. Let $p_{\text{old}}(x)$ be the previous router's distribution over the old experts for a token $x$ from the replay set $\mathcal{R}$. Let $\tilde p(x)$ be the new router's distribution, renormalized over those same old experts. Let $\mathcal{L}_{\text{new}}$ be the detection-and-explanation loss on the new family, with the same dual objective as MOS. The stage objective is

$$\mathcal{L} = \mathcal{L}_{\text{new}} + \beta \, \mathbb{E}_{x \in \mathcal{R}} \, \mathrm{KL}\big(p_{\text{old}}(x) \,\|\, \tilde p(x)\big),$$

where $\beta$ trades how quickly the new family is learned against how stable old routes remain. This KL term is the key ingredient. Without it, new experts can take over top-$k$ slots that old-family tokens depend on.

After each stage, χFT's learnable merge is applied to produce a dense release model, and both artifacts are evaluated. The intended outcome is either a measured trade-off curve between update cost, backward stability, and new-family accuracy, or a clean negative result showing that dense continual learning suffices.

Baselines

The naive reference is the stage-1 model left unchanged. This "do nothing" model shows how much each new family actually requires new capacity, as opposed to falling out of general code knowledge.

The rule-based references are Slither for the classic classes and CDRipper for centralization defects. CDRipper reports 93.7% overall precision (Lin et al.), and the learned auditor must be compared against it on the hand-verified subset. Because CDRipper also produced the training labels, we report agreement and disagreement separately. We especially examine disagreements on contracts whose owner is a multisig or DAO, which is where Lin et al. report CDRipper's false positives.

The strongest learned competitor is a dense model with the same base and the same data, retrained from scratch at every stage on the union of all families. This is the accuracy ceiling a practitioner would reach for. For the classic classes, the best reported learned result in the file is MOS's average gain of +6.32% F1 and +4.80% accuracy over 16 baselines (MOS). The file gives no absolute MOS F1, so we re-run MOS's recipe ourselves as a second competitor. That re-run is an MoE retrained jointly at every stage.

The strongest continual competitor is a dense model fine-tuned sequentially with the same replay set plus distillation on its own previous predictions. Its role is to isolate whether modularity matters at all, rather than replay alone.

All learned systems get the same data and the same replay budget. Each also gets an equal hyperparameter search budget, set at a fixed number of trials. GPU-hours are reported per stage.

Ablations remove one ingredient at a time. The variants are: unfreezing the shared experts, dropping the routing-preservation term, and appending new experts without freezing the old ones.

The proposal beats the competitors under a specific condition. It must match joint retraining on new-family F1 within about one point. It must also show clearly smaller old-family F1 loss and fewer negative flips than every continual baseline. A negative flip is an old-family bug the previous release caught and the new release misses. Finally, its per-stage training cost must be a fraction of joint retraining.

Why not just…?

Why not just use static analyzers for the new bug families? For upgrade and centralization defects, this is a serious option. CDRipper's precision is high, and Li et al.'s USCDetector reaches 96.26% accuracy at identifying upgradeable contracts. The file reports two weaknesses. CDRipper analyzes each contract in isolation, which produces false positives when ownership is held by an external DAO or multisig. And every new pattern needs hand-written rules. We expect the learned auditor to win mainly on those context-dependent cases. If it does not beat CDRipper on the verified subset, we will report that stage 2 adds nothing beyond the tool. That would not break the continual-learning result on stage 3, where no comparable rule-based defect detector is reported.

Why not just retrain a dense model on everything each time? This is the main threat. If retraining is cheap enough and its negative flip rate is low, the MoE structure is unnecessary. The joint-retraining baseline answers this directly. The file contains no reported measurement of verdict churn under retraining for smart-contract detectors, so the question is open rather than settled in our favor.

Why not just prompt a frontier model with the new bug descriptions? Prompting requires no retraining, but reported real-world performance is weak. LLM-SmartAudit falls to 47.6% accuracy on real projects even with role-specialized agents (LLM-SmartAudit). On EVMbench, the best unassisted DETECT recall is 45.9%, from Claude Opus 4.6. We include a GPT-4o-class prompted baseline at every stage with the same class descriptions.

Why not just train one adapter per bug family plus a family classifier? This is close to our design, with coarser routing. If it matches our results, the contribution reduces to "freeze and append." We would say so, and the token-level router would be shown to be unnecessary. It runs under the same budget.

Proposed Experiments

The main experiment runs the three-stage sequence. For each class, we report per-class F1, change in old-family F1 (backward transfer), negative flip rate on old-family test contracts, GPU-hours per stage, and serving memory for both the MoE and the merged model.

The first decisive check targets the continual baseline. If dense replay plus distillation matches our backward transfer and flip rate at similar cost, the core claim fails.

The second check targets merging. We test whether χFT merging preserves old-family recall in this setting. If merging reintroduces the forgetting the MoE avoided, the "extend the MoE, serve the merge" story fails, and only the MoE can be served.

The third check is a transfer test. The stage-3 model's flagged functions and predicted bug families are given as hints to an agent on EVMbench DETECT. The comparison point is the 45.9% unassisted recall, motivated by the reported jump from about 39% to over 90% PATCH success when a mechanism hint is supplied. Training data must exclude EVMbench's Code4rena repositories, which date from July 2023 onward. This check is secondary. A null result here would not overturn the main claim.