Semantic Balance Trees for Scalable, Interpretable Few-Shot Text Classification

Interpretable text classifiers that explain decisions in plain language usually need a written description of how each pair of classes differs ("how do I tell a lost card from a card that hasn't arrived?"). With $K$ classes there are $K(K-1)/2$ such pairs. That is manageable for four news topics and hopeless for 77 banking intents. Compositional data analysis, the statistics of proportions that must sum to one, solved the same counting problem long ago. Any distribution over $K$ categories can be described exactly by $K-1$ "balances," each a log-ratio contrast between two groups of categories. If the groups come from a tree, these contrasts are orthogonal, so the description loses nothing and repeats nothing. We propose to build that tree from the meanings of the class labels, attach one natural-language contrast to each of its $K-1$ balances, and read the classifier's probabilities directly off those balance scores.

In two sentences: cluster the label descriptions into a semantic tree, and give each internal split one LLM-written contrast concept whose text-similarity score is a coordinate in an orthonormal log-ratio basis. The classifier then needs $K-1$ concepts instead of $K^2/2$, and every pairwise decision decomposes exactly into the few contrasts on the tree path between the two classes.

Core Claim

The isometric log-ratio (ILR) transform maps a probability vector over $K$ classes to $K-1$ unconstrained coordinates in an orthonormal basis $\Psi$, so that ordinary Euclidean distance in those coordinates matches the natural "Aitchison" distance between distributions. Tree-Structured Orthonormal Decomposition of the Aitchison Simplex (PolyILR) constructs such a basis for any tree, including nodes with three or more children, using local Helmert contrasts. Each contrast compares one child group against its remaining siblings. The paper also shows that a softmax classifier's logit space is isomorphic to the space these coordinates live in. Our model outputs one balance score $z_m(x)$ per basis row $\psi_m$, for $m = 1, \dots, K-1$, and sets the centered log-probabilities to $z(x)\Psi$. For any two classes $i$ and $j$,

$$\ln\frac{p_i(x)}{p_j(x)} = \sum_{m=1}^{K-1} z_m(x)\,\big(\Psi_{mi} - \Psi_{mj}\big),$$

and $\Psi_{mi}$ is nonzero only when class $i$ lies below the node that balance $m$ splits. The odds between two intents are therefore explained exactly, with no residual, by the handful of balances along their tree path. Because the basis has full rank $K-1$, this is a reparameterization of softmax rather than an approximation, so no expressiveness is lost. What changes is that every logit coordinate now has a name.

We are deliberately not claiming two benefits the research file lists. The first is subcompositional coherence under candidate pruning. Pairwise log-ratios of a softmax over dot products are already differences of logits, so they are already invariant to which other candidates are present. The second is that making label vectors an orthonormal basis would by itself change predictions. It would not: centering and rotating logits leaves the softmax unchanged. The basis matters only when each coordinate is tied to a separately computed, nameable signal, which is exactly what we build.

Research Program

The pipeline has four steps.

Build the tree. Embed each class name and short description with a sentence encoder and run agglomerative clustering. Cut merges whose heights fall within a tolerance into a single node with many children, so the tree keeps genuine polytomies. PolyILR shows that forcing these into arbitrary binary splits destroys stability, with feature-selection Jaccard dropping to near 0 versus 0.43 to 0.92 for its polytomous basis.

Build the basis. Construct $\Psi$ with PolyILR's leaf-count-weighted Helmert contrasts.

Write one contrast concept per balance. For each balance, prompt an LLM with the class descriptions on its two sides ("group A versus groups B∪C") and ask for a short set of contrastive statements per side. This takes roughly $K-1$ calls, plus refinement calls in the style of the Generate-Predict-Refine loop from Adaptive Concept Discovery for Interpretable Few-Shot Text Classification (StructCBM).

Score and fit. Compute $z_m(x)$ as the difference between the input's mean cosine similarity to the "+" side statements and to the "−" side statements. Multiply by a per-balance scale and add a bias. Fit only these $2(K-1)$ scalars by cross-entropy on the few labeled examples, with a frozen encoder. In the zero-shot setting, use one shared scale and no bias.

The analysis has three parts. First, we test whether the semantic tree matters for accuracy, or only for interpretability, by comparing it with random trees of identical shape. Second, we check the exactness of explanations: deleting balance $m$'s concept should shift each pairwise log-odds by exactly the amount the equation above predicts, and we measure how well that predicted shift tracks the change in the model's actual decision. Third, we measure tree stability, a known open problem since PolyILR assumes a fixed tree. We rebuild trees under different encoders and different paraphrases of the label descriptions, and report Jaccard overlap of the resulting splits along with accuracy variance.

Relation to Prior Work

StructCBM motivates the proposal. It shows that LLM-written prototypical and discriminative concepts, matched by embedding similarity, give accurate and faithful few-shot classifiers. It reports 85.45% 10-shot accuracy on AGNews, 173 API calls, and a 4.39/5 faithfulness rating, and it names the quadratic growth of pairwise concepts as its scaling limit. We replace its pairwise library and heuristic score sums with $K-1$ contrasts whose combination rule is fixed by the geometry.

PolyILR supplies the basis construction. It applied the basis only post hoc, to diagnose CIFAR-100 gradients on a given taxonomy, and left training with such decompositions open. We use the basis as the model's output layer and derive the tree from language.

Three other ILR papers show the coordinates are trainable and interpretable but stay label-agnostic:

BTZSC constrains our ambitions. It finds that rerankers lead zero-shot accuracy, while embedding models are the efficiency sweet spot. We therefore target the embedding regime and do not expect to beat rerankers on raw zero-shot accuracy.

Baselines

Plain dual encoder. The naive reference point is a frozen GTE-large-en-v1.5 encoder, the embedding model BTZSC singles out for efficiency, with a softmax over cosine similarities to the BTZSC-style verbalized label names. In the few-shot setting it is given the same $2K$-scalar calibration budget. It has no concepts. If our model cannot match it, the concepts are costing accuracy.

StructCBM. This is the strongest existing interpretable competitor. On AGNews 10-shot it reports 85.45% accuracy with 173 API calls. For SST2, MedAbs and FinaQuery, no reported accuracy was found on Lacuna. We rerun it with the same encoder, the same LLM, and the same 10 shots and seeds. On Banking77 we run it twice: with its full pairwise library if affordable, and in a budget-matched variant restricted to the nearest-neighbor pairs that fit our API budget. Beating it means matching its accuracy within one point on the four-class tasks with no more API calls, and exceeding its budget-matched variant on Banking77.

Strongest zero-shot system. We report Qwen3-Reranker-8B as an upper reference. BTZSC reports a macro F1 of 0.72 for it over all 22 datasets. No per-dataset Banking77 number was found on Lacuna, so we rerun it there. We do not expect to beat it. We report the accuracy-per-inference-cost gap.

Ablations. Three ablations isolate the key ingredient:

  • A random tree with the same shape and concepts regenerated for its splits isolates the role of semantics.
  • A binarized tree tests PolyILR's polytomy claim in this new setting.
  • Balances fitted as linear probes on raw embeddings, with no concepts, isolate whether the written concepts add accuracy or only explanation.

All methods share data, seeds, encoder, and an equal hyperparameter-search budget.

Why not just…?

Why not just keep StructCBM and generate concepts only for the most confusable pairs? This is the main threat, since keeping roughly $m$ neighbors per class gives about $Km$ concepts. It loses two things. There is no guarantee that a distant pair the model happens to confuse has any concept at all. And the combined score is still an ad hoc sum, so per-pair explanations are not exact. The budget-matched StructCBM run settles this. If it ties us on Banking77 accuracy and faithfulness, our contribution shrinks to the exactness of the decomposition, and we will report it that way.

Why not just use a hierarchical classifier, with a local softmax at each tree node multiplied down the path? This nested-softmax model also uses about one decision per internal node. However, its log-probabilities are not linear in the node scores, its node decisions are not orthogonal, and an error at the root caps the probability of every class beneath it. We run it on the same semantic tree with the same concepts per node. If it matches our accuracy, the orthonormal basis buys only the additive, exact attribution, which our concept-deletion test measures directly.

Why not just use semantic soft targets with ordinary cross-entropy? Spreading the target probability toward semantically similar classes is the natural fix for Exact-ILR's uniform smoothing, and it needs no log-ratio machinery. That is exactly why we do not make it our contribution. It would likely work about as well as an ILR-specific version, and it does not address the concept-count problem. We include it only as an optional regularizer, applied identically to every baseline.

Proposed Experiments

We run 10-shot experiments on AGNews, SST2, MedAbs and FinaQuery to match StructCBM's protocol. Banking77 is the scaling test, run 10-shot and zero-shot, with 77 classes giving 76 balances versus 2,926 pairs. We use five seeds. We report accuracy and macro F1, expected calibration error (where Exact-ILR reports gains from ILR-space modeling), LLM API calls, number of concepts, and inference cost.

For faithfulness, the deletion test compares the predicted change in log-odds against the actual change in the decision, and we collect a small human rating on StructCBM's scale. Tree stability is reported as Jaccard overlap of splits across encoders and description paraphrases.

Three outcomes decide the proposal:

  1. On Banking77 10-shot, the model must come within one point of the best dual-encoder baseline while using at most about $2K$ API calls. If it fails, the bottleneck is too restrictive in practice.
  2. The semantic tree must beat random trees of the same shape on accuracy or faithfulness. If it beats them on neither, label semantics are not doing work.
  3. Removing one balance's concept must change pairwise decisions only for class pairs whose tree path crosses that balance. If changes leak to other pairs, the claimed exact, path-local explanation does not hold in practice.