Which Experts Override the Evidence? Separating Vagueness from Fabrication in Retrieval-Augmented Mixture-of-Experts Models

Language models given retrieved documents still fail in two different ways. Sometimes they say too little: asked to describe a leopard with a passage about rosettes and tree-hoisted kills in front of them, they answer "a large animal found in Africa and Asia". The answer is true but generic. Other times they say too much, adding specific details that the passage never mentions. Research on generic outputs and research on fabricated outputs rarely talk to each other, and fixes for one tend to make the other worse. A model pushed to commit to fine-grained attributes invents some of them. A model pushed to stay faithful retreats to safe category words.

Mixture-of-experts (MoE) models offer a way to pull these two failures apart. An MoE replaces each feed-forward block with many parallel sub-networks, called experts, and a small gating network (the router) picks which experts process each token. Many MoE models also contain "shared" experts that every token passes through. Interpretability work on such models finds that the shared experts handle coarse knowledge, such as recognising that Canada is a country, while the routed experts handle specific facts, such as linking Canada to Ottawa. Separate work on dense models finds that retrieval-grounded hallucinations happen when late feed-forward layers inject the model's internal knowledge strongly enough to drown out evidence the attention heads have already copied from the context. We propose to test whether the two failures map onto the two kinds of expert. If they do, the trade-off between being specific and being faithful can be controlled separately for each expert type, which is not possible when all feed-forward layers are treated as one block.

Core Claim

Our hypothesis has three parts. In retrieval-augmented MoE models with shared experts, under-specific answers come mainly from shared-expert outputs that override differentia (the distinguishing attributes) present in the retrieved passage. Baseless fabricated details come mainly from routed-expert outputs that add specific content the passage does not support. A token-level intervention that scales shared and routed contributions independently, conditioned on whether the context supports a specific claim, should therefore reach a better joint frontier of specificity and faithfulness than any uniform down-weighting of feed-forward output.

The claim can be falsified. If the per-token contributions of the two expert types do not differ between the two failure types, or if uniform scaling reaches the same frontier, the expert split adds nothing and we will report that.

Relation to Prior Work

Two mechanistic papers motivate the hypothesis. ReDeEP separates a context score, computed from the attention heads that copy context into the output, from a parametric score, computed from late "knowledge" feed-forward layers. It shows that in roughly 65–77% of RAG hallucinations the attention heads attended correctly and the feed-forward layers overrode them. Its steering method, AARF ("add attention, reduce FFN"), boosts the copying heads and dampens those layers. ReDeEP studies only dense LLaMA models. It also counts only divergence from the context as failure, so a vague but true answer counts as a success. Decoding Knowledge Attribution in Mixture-of-Experts establishes the basic-versus-refinement division of labour between shared and routed experts in Qwen 1.5-MoE, OLMoE and Mixtral. That work is closed-book and proposes no intervention. We join the two: ReDeEP's parametric score, computed separately for each expert type, under retrieval.

Three papers connect MoE to retrieval or hallucination at a different level from ours. MIXRAG places experts in the graph retriever and leaves the generator a frozen dense model. RAMoLE retrieves LoRA adapters (small add-on weight updates) rather than documents. MoLE treats internal layers of a vision-language model as "experts" during decoding, but those are layers, not parallel sub-networks. None of the three asks which parameter experts inside the generator override the evidence.

The work on generic outputs supplies our measurements and our strongest specificity baselines. Definition Modelling for Appropriate Specificity re-ranks beam candidates with two scores. Its under-specificity score asks whether the definition could be produced from the word alone; its over-specificity score asks whether the definition leaks the context sentence. Fine-grained Contrastive Learning for Definition Generation trains with a contrastive loss that aligns word and definition representations and lowers the share of under-specific errors. LM-Lexicon builds a sequence-routed MoE from domain-specific definition experts. All three are closed-book and none measures fabrication. That is a real limit: they reward specificity without checking whether the specific content is supported.

RAGTruth provides annotations of hallucinated spans. Its finding that models add baseless information far more often than they contradict the source is the reason we treat fabricated differentia as a separate category from contradiction.

Research Program

The first component is a controlled evaluation set, because existing benchmarks measure either specificity or faithfulness, never both. We take target terms from Oxford, WordNet and Urban Dictionary, plus a held-out set of terms coined after each model's training cutoff. For each term we pair the definition task with a retrieved passage under three conditions. In the first, the passage contains the hypernym (the category word) and at least two differentia. In the second, it contains only the hypernym. In the third, it contains differentia that conflict with what the model likely believes. We extract differentia from the gold definition and the passage, align them with an entailment model, and spot-check the alignment by hand.

Each output is then scored on two axes. Specificity is the fraction of supported differentia that the output expresses. We add the under-specificity score of Definition Modelling for Appropriate Specificity and the WordNet depth of the head noun in the output. Fabrication is the fraction of expressed attributes that the passage does not support, measured with a RAGTruth-style span detector. Separating condition one (the evidence is there; vagueness is an override) from condition two (any specific detail is fabricated) is what lets us tell the mechanisms apart. We check that the protocol transfers by also running it on the question-answering and summarisation portions of RAGTruth.

The second component is attribution per expert type. For each generated token at layer $\ell$, we compute the parametric contribution of the shared experts, $P^{s}_\ell$, and of the routed experts, $P^{r}_\ell$. Each is the Jensen–Shannon divergence between logit-lens next-token distributions (the distributions obtained by decoding intermediate hidden states directly into vocabulary) taken before and after adding that expert group's output to the residual stream. This mirrors ReDeEP's parametric score but splits it by expert type. We also compute ReDeEP's context score for the copying heads.

The prediction is directional. Hypernym-substitution tokens in condition one should show high $P^{s}$ relative to the context score. Fabricated-differentia tokens in condition two should show high $P^{r}$. Qwen 1.5-MoE has shared experts and is the main subject. OLMoE and Mixtral have no shared experts and serve as structural controls. In them we test whether a stable subset of always-frequently-routed experts plays the generalist role. If no such subset exists, the hypothesis should hold only where shared experts exist.

The third component is a decoding-time intervention. At layer $\ell$ we rewrite the residual update as

$$h_\ell = h_{\ell-1} + \beta\, a_\ell + \alpha_s\, f^{s}_\ell + \alpha_r \sum_i g_i\, f^{r}_{\ell,i},$$

where $a_\ell$ is the attention output, $f^{s}_\ell$ is the shared-expert output, $f^{r}_{\ell,i}$ is routed expert $i$'s output, and $g_i$ is its router weight. The scales $\beta$, $\alpha_s$ and $\alpha_r$ are set per token by a small linear probe. The probe reads the context score and the two parametric scores and predicts two risks: that the token is an override of available differentia, and that it is an unsupported specific detail. When differentia are available and shared-expert dominance is high, the probe lowers $\alpha_s$ and raises $\beta$. When the context gives no support and routed-expert dominance is high, it lowers $\alpha_r$. We train the probe on a split of our evaluation set and test on held-out terms and on RAGTruth. The intervention requires no fine-tuning of the generator.

Baselines

The naive reference point is the same MoE model given the same retrieved passage under greedy and beam decoding. We also run it with an instruction to include distinguishing attributes and use only the passage, so that prompting effort is not confounded with mechanism.

The strongest existing intervention is AARF from ReDeEP, ported to the MoE model. It treats every expert output in the late layers as one feed-forward block and scales it down uniformly when its own detector fires. We give it the same layer-search and threshold-tuning budget as our probe, on the same training split. It is the closest competitor because it already exploits the attention-versus-feed-forward tug-of-war; our proposal differs only in splitting the feed-forward side. ReDeEP reports a detection AUC of 0.7458 on RAGTruth with LLaMA2-7B, against 0.7290 for RAGAS. No reported result was found on Lacuna for ReDeEP or AARF on any MoE model, so we reproduce both ourselves.

The key ablation tunes a single scale shared by both expert types against our two separate scales, holding the probe's inputs fixed. Beating it means a strictly better frontier of specificity against fabrication: at matched fabrication rate, at least five points higher differentia recall in condition one, with the confidence interval excluding zero across three seeds of probe training. A second ablation swaps the probe's targets, using shared-expert signals to predict fabrication and routed-expert signals to predict vagueness. If the hypothesis is right, this swap should do worse.

On the specificity side we compare against Definition Modelling for Appropriate Specificity's dual re-ranker, applied to beam candidates from the RAG-conditioned MoE model and given its weights tuned on our training split. In its original closed-book setting, that paper reports Oxford BLEU rising from 18.57 to 26.52. Fine-grained Contrastive Learning for Definition Generation reports an under-specific error ratio on Oxford of 4.8%, down from 7.6% for T5-Base. These numbers are not directly comparable to our retrieval-conditioned setting. They give a reference for how much specificity training alone buys, and we report our under-specific error rate on the same Oxford split.

On the fabrication side we pair any method with response filtering by a fine-tuned RAGTruth detector. RAGTruth reports a span-level F1 of 52.7% for its fine-tuned Llama-2-13B detector against 28.3% for GPT-4, and a 63.2% reduction in Llama-2-7B's hallucination rate from filtering.

Why not just…?

Why not just apply AARF uniformly to all experts? This is the real threat. If shared and routed experts both contribute to both failures in proportion, uniform down-weighting reaches the same frontier and our split is unnecessary. We expect uniform damping to trade the failures against each other. Removing parametric push should cut fabrication but leave the model unable to assemble the differentia the passage offers, or push it towards copying context verbatim, which the over-specificity score detects. The tied-scale ablation decides this directly. If it matches our method within noise, we will conclude that the useful signal lies in the attention-versus-feed-forward balance alone, as ReDeEP proposed.

Why not just prompt a frontier model to include distinguishing attributes from the passage? Prompting is cheap and must be beaten. The evidence in the file suggests it is not sufficient on either axis taken separately. In LM-Lexicon, human-rated definition accuracy was 0.80 for Claude-3-Opus and 0.77 for GPT-4-Turbo, against 0.85 for the specialised MoE. In RAGTruth, GPT-4 managed only 28.3% span F1 at spotting unsupported content, which suggests it is also weak at avoiding such content. Still, neither result measures prompting on our joint objective. We therefore include instruction prompting as a baseline on the same MoE and report a frontier-model API run on the evaluation set, without internal analysis.

Why not just generate candidates and re-rank for specificity and faithfulness? A dual re-ranker combined with a hallucination filter may approach our frontier on in-distribution terms. Its weaknesses are cost and how the weights carry over. Definition Modelling for Appropriate Specificity needs three model passes and weights tuned per dataset. Re-ranking also can only choose among candidates the decoder already produced; if every beam falls back to the hypernym, there is nothing specific to pick. We test this on the post-cutoff terms and on Urban Dictionary, after tuning the re-ranker on Oxford. We report both quality and wall-clock cost per output. If re-ranking matches us at comparable cost on the held-out sets, our contribution reduces to the mechanistic finding alone.

Why not just retrieve better passages? A better retriever raises the ceiling for both methods but leaves the failure on the decoder side untouched. ReDeEP's finding that attention is correct in most hallucinations means the evidence was already present and was overridden. Condition one of our set holds the evidence fixed and correct by construction, so retrieval quality cannot explain any gap observed there.

Proposed Experiments and Decisive Checks

The first decisive check is the attribution result in Qwen 1.5-MoE. Take hypernym-substitution tokens in condition one and fabricated-differentia tokens in condition two. Does the ratio of shared to routed parametric contribution separate them, with a token-level classification AUC well above what the pooled parametric score achieves? A null here ends the intervention arm.

The second check is causal. Ablate the shared experts at the identified layers on condition one. Ablating the shared experts should increase differentia recall there. Ablating the top routed experts on condition two should reduce fabrication without a matching loss of recall in condition one. The two effects should differ in sign across the conditions. The paper on knowledge attribution in MoE reports that blocking the top experts cut mean reciprocal rank by 43% in Qwen 1.5-MoE and by 76% in OLMoE. We therefore use partial scaling rather than full ablation and track perplexity on held-out text, to separate targeted effects from general breakage.

The third check is the frontier comparison on the controlled set and on RAGTruth. Here we compare our probe against uniform AARF, the tied-scale ablation, the re-ranker plus filter, and instruction prompting, all at matched tuning budgets. We report differentia recall, under-specific error rate and BLEU on Oxford, and fabrication rate and span-level hallucination rate on RAGTruth.

The fourth check uses OLMoE and Mixtral as controls. If an effect appears there with no identifiable generalist experts, we learn that the explanation is not about shared experts specifically. We would revise the claim to concern routing confidence in general.