Shrinking a language model to 2–3 bits per weight is the most direct way to cut its memory and serving cost. The usual checks are perplexity and broad benchmarks, and those mostly reward fluent, general-knowledge behavior. We suspect they hide a specific failure. Compression may first erase narrow, specific facts, the leaves of a topic hierarchy, while keeping broad domain knowledge and fluency intact. The compressed model would then answer confidently and wrongly about exactly the niche questions where a user cannot tell. This proposal measures that failure and tests one mechanism for preventing it. The weights a model needs for a given fact are shared with facts that sit nearby in a topic tree, so a calibration set built to cover the branches of that tree should protect factual knowledge more cheaply than a calibration set sampled at random.
The core idea fits in two sentences. Build a topic hierarchy over the knowledge a model is expected to hold, and test whether the weights that matter for answering a question overlap more for questions close together in that hierarchy. If they do, choose the small set of examples used to decide which weights stay at full precision by covering the tree's branches. The quantized model then keeps more specific facts at the same bit budget and the same calibration cost, and hallucinates less on them.
Core Claim
We make two claims, and each can be tested on its own. The first is a measurement claim. At equal average bits, accuracy loss and confident-wrong answers under low-bit quantization grow with the depth of a question's topic in a corpus hierarchy, even after controlling for entity popularity. Depth here means how specific the topic is: the root is broad background vocabulary and the deep nodes hold niche terms, as in the nested tree prior of Blei et al. and the geometric picture of Meng et al., where specific terms sit at the periphery of an embedding sphere.
The second is a mechanism claim. Let $M_q$ be the set of weights a saliency method marks as critical for question $q$, and let $d_T(q,q')$ be the path distance between the leaves of $q$ and $q'$ in a topic tree $T$. We claim the overlap $J(q,q') = |M_q \cap M_{q'}| / |M_q \cup M_{q'}|$ decreases with $d_T$. We further claim that tree distance predicts this overlap better than distance under a flat clustering with the same number of clusters. If both claims hold, a fixed budget of calibration examples spread across tree branches covers more of the model's fact-bearing weights than the same budget drawn at random or from flat clusters.
Relation to Prior Work
The method we build on is Task-Circuit Quantization (TACQ). TACQ uses gradient-based saliency on a task-specific calibration set to find the small fraction of weights to keep in 16-bit, and it quantizes the rest aggressively. Its results are strong on task benchmarks, but its "circuits" are defined per task (MMLU, GSM8k, Spider). It does not ask whether quantization changes hallucination rates differently from perplexity, and it notes that gradient capture on calibration data is costly. We keep TACQ's preservation mechanism and change two things: what the calibration set covers, and what the evaluation measures.
Three pieces of prior work motivate the depth hypothesis. HaluEval reports that hallucinations concentrate in particular topics. TopicMistral finds that suppression of hallucinated topics happens mainly in deeper layers. That suggests the relevant weights are localized and not spread uniformly, which is a precondition for selective preservation to matter.
The tree construction comes from two sources. HEAL builds topic trees with hierarchical non-negative matrix factorization and shows that respecting the hierarchy cuts retrieval false positives from 0.8534 to 0.0878. HiCOT builds hierarchies cheaply from lightweight document vectors with agglomerative clustering. That matters here because tree construction must cost little compared with the gradient passes it is meant to save.
Our work differs from all of these. HEAL and HiCOT never touch the generator's weights. TACQ never looks at topic structure or factual error. The hierarchy-and-hallucination papers, SeSE and TopicMistral, use hierarchy either as a post-hoc uncertainty score over ten sampled answers or as the output format of the task, not as a tool for compression.
Research Program
The first step is to build the knowledge tree. We start from the Wikipedia passages linked to the entities in PopQA and TriviaQA and from the subject structure of MMLU. Over these we build a three-to-four-level tree with HNMFk, following HEAL, and a second tree with HiCOT-style agglomerative clustering, so that no conclusion depends on one tree-building method. Each evaluation question is attached to the leaf of its source passage. As a sanity check against arbitrary trees, we verify that the HAC tree recovers MMLU's own subject grouping well above chance.
The second step is the overlap study, which is the mechanism test and the part most likely to fail. On Llama-3-8B-Instruct, the model TACQ reports on, we compute TACQ's saliency masks for single questions or small groups of questions sampled across the tree. We then regress mask overlap on tree distance, with controls for lexical overlap between questions, embedding cosine similarity, and log entity popularity (PopQA provides popularity). If tree distance has no predictive power beyond embedding similarity, the hierarchy is decoration and we report that negative result.
The third step is the calibration procedure. With a fixed calibration budget of $K$ examples, we allocate examples top-down. Each internal node receives examples in proportion to the number of descendant leaves it covers, which avoids allocating by question frequency, and the procedure ensures every branch down to a chosen depth gets at least one example. Saliency masks are combined across examples exactly as TACQ does, and the same fraction of weights is kept in 16-bit. The procedure costs no more gradient passes than TACQ with $K$ random examples. The only addition is clustering text that is already available.
The fourth step is the depth-stratified evaluation. We report factual accuracy, and separately confident-wrong rate, as a function of tree depth and popularity bin. The confident-wrong rate counts answers that are wrong and given without hedging when the model is allowed to say it does not know.
Baselines
The naive reference points are the unquantized 16-bit model, which is the ceiling, and TACQ calibrated on its original general-purpose data. The latter shows what current practice does to specific facts.
The strongest competitor is TACQ with a calibration set of $K$ factual QA examples drawn uniformly at random from the same pool. The same pool, bit budget, preserved-weight fraction, gradient-pass count, and seeds are used for every method, and tuning gets the same grid. The strongest reported number for this setting comes from TACQ on Llama-3-8B-Instruct: 96% of unquantized MMLU performance recovered at 3.1 bits, a 5.25-point absolute gain over SPQR. At 2 bits, TACQ reports an average 14.74% gain over SliM-LLM. No reported result was found on Lacuna for hallucination or PopQA accuracy of any quantized model, so the depth-stratified numbers for this baseline will be measured here. Beating it means two things at matched bits and matched calibration cost. First, statistically significant gains on deep-leaf and low-popularity PopQA questions and on FActScore. Second, no loss of more than one point on MMLU recovery or WikiText2 perplexity.
Two ablations isolate the hierarchy. The first stratifies calibration by a flat k-means clustering with as many clusters as the tree has leaves. This removes the nesting but keeps the coverage. The second stratifies by entity-popularity bins. That tests whether "specific" just means "rare". A third ablation applies tree stratification only at the top level, which shows whether depth of coverage matters.
Why not just…?
Why not just stratify the calibration set by entity popularity, since rare entities are the long tail anyway? This is the most serious threat, because tree depth and popularity are correlated. The overlap study addresses it directly by including popularity as a control. The popularity-stratified ablation addresses it again on the actual compression outcome. If popularity stratification matches tree stratification on deep-leaf accuracy, we drop the hierarchy claim and keep the measurement result. The reason we expect a difference is that popularity says nothing about which rare facts share weights. Ten rare facts from one branch may all reuse one circuit while a rare fact from an unsampled branch stays unprotected.
Why not just use flat clusters or embedding similarity instead of a tree? The flat-cluster ablation answers this. Flat clusters spread examples evenly across fine-grained groups but give no guarantee that broad branches are covered, and they discard the parent–child sharing we hypothesize. HiCOT's evidence that hierarchical clustering beats flat neural topic models on coherence is suggestive but concerns topic quality, not weights, so the experiment has to decide this.
Why not just quantize less aggressively, say 4 bits? That gives back much of the memory-bandwidth saving that motivates compression. The measurement study reports depth curves at 2, 3, and 4 bits. If deep-leaf loss vanishes at 4 bits, that tells practitioners when the problem matters. It does not remove the question for the 2–3-bit regime, where TACQ shows standard methods collapse.
Why not just run a cheap hallucination detector on the compressed model's outputs? Cost-Effective Hallucination Detection shows that two or three calibrated single-generation scores reach 0.9106 F1 on TriviaQA against 0.8614 for SelfCheckGPT. Detection flags errors but does not restore lost facts. Its calibration was also fit on unquantized models. As a side check, we test whether the self-assessed probability that an answer is true stays calibrated on deep leaves after quantization. If it does, detection plus abstention is a real cheaper alternative for the hallucination part of the problem, and we would say so.
Proposed Experiments
The decisive experiment comes first. Before any quantization, the overlap regression must show a significant negative coefficient on tree distance after controls, replicated across both tree builders. Without this, the method has no basis.
Second, the depth-degradation curves for TACQ with random calibration at 2 and 3 bits on PopQA, TriviaQA, and FActScore must show that accuracy loss grows with depth after controlling for popularity. Without this, there is no problem to solve.
Third, at each calibration budget $K$, we compare tree-stratified calibration against random, flat-cluster, and popularity-stratified calibration on deep-leaf accuracy and confident-wrong rate, with MMLU recovery and WikiText2 perplexity as guards against regressions elsewhere. The headline result we hope for is calibration efficiency: tree stratification reaching random-calibration accuracy with a fraction of the gradient passes, which addresses the calibration-cost limitation TACQ itself notes. We will repeat the main comparison on Qwen2.5-32B-Instruct, where TACQ reports 87.93% MMLU recovery at 2 bits, to check that the effect holds as models grow.
References
- Hierarchical Topic Models and the Nested Chinese Restaurant Process
- Discriminative Topic Mining via Category-Name Guided Text Embedding
- Task-Circuit Quantization: Leveraging Knowledge Localization and Interpretability for Compression
- HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
- Addressing Topic Granularity and Hallucination in Large Language Models for Topic Modelling
- HEAL: Hierarchical Embedding Alignment Loss for Improved Retrieval and Representation Learning
- HiCOT: Improving Neural Topic Models via Optimal Transport and Contrastive Learning
- SeSE: A Structural Information-Guided Uncertainty Quantification Framework for Hallucination Detection in LLMs
- Cost-Effective Hallucination Detection for LLMs