Do African-Language Benchmark Rankings Survive Fair Tuning?

Most published comparisons on African-language benchmarks fine-tune every pretrained model with one fixed recipe: the same learning rate, batch size and number of epochs. The leaderboards built this way support a widely repeated conclusion, that Africa-centric encoders beat massively multilingual ones. Nobody has checked whether that conclusion survives giving every model a fair share of hyperparameter search. Work on character-level morphology has already shown that the cheaper fix can mislead. Under one shared recipe Transformers looked better than LSTMs, and once both were searched properly the ordering reversed (Wiemerslage et al.). Low-resource languages make the problem worse in two ways. Small validation sets make every hyperparameter choice noisy, and tuning dozens of model–language pairs costs more than most groups working on these languages can spend.

We propose two things. The first is a standardized re-evaluation of AfriSenti and MasakhaNER 2.0 in which every model gets the same, logged tuning budget, and we report how sensitive each model is to its settings alongside its best score. The second is a cheap test, run before any search, that predicts which model–language pairs need a large search. The test borrows an idea from physics. There, the vibration modes of an operator can either spread evenly over a domain or "localize", meaning they pile into a small region. We look at the directions in which the fine-tuning loss curves most sharply and ask whether they spread across the whole network or concentrate in a few blocks of weights, such as the attention query and key matrices. Our hypothesis is that concentrated curvature means a narrow window of good learning rates. A narrow window is exactly the case where a single default recipe fails and makes a model look worse than it is.

Core Claim

We make two claims. Each can fail independently of the other.

The first claim is empirical. On AfriSenti and MasakhaNER 2.0, a non-trivial share of the published model orderings are artifacts of untuned defaults. The share is largest when training data is scarce, and fair, budget-matched tuning changes it measurably. This would mirror what BARS found for recommender systems and what Locatello et al. found for disentanglement, but in a setting where it has not been tested.

The second claim is mechanistic. How concentrated the top curvature directions are, measured before training, predicts each model–language pair's sensitivity to its hyperparameters. It predicts better than the size of the largest curvature value, a quick learning-rate sweep, or the tokenizer's word-splitting rate do on their own.

We frame the second claim around localization, and not merely around sharpness, for a theoretical reason. Ormaniec et al. show that in self-attention the curvature of the query and key weights grows with the sixth power of input magnitude, while the curvature of the value weights grows only with the second power. Small shifts in input statistics should therefore move curvature between parameter blocks, not just raise or lower it overall. Tokenizer fragmentation is one such shift: on morphologically rich languages it lengthens sequences and changes token statistics. If fragmentation pushes curvature into a few blocks, then one global learning rate will suit those blocks or the rest of the network, but not both.

Research Program

The evaluation harness fixes everything except the model and its hyperparameters. It fixes data loading and splits through the Hugging Face Datasets library. It also fixes text preprocessing, the label scheme, metric code, the number of random seeds, and model selection on the validation split only. We include this level of control because NerfBaselines showed that trivial pipeline differences can reorder a leaderboard.

The model pool is mBERT, XLM-R base and large, AfriBERTa, AfroXLMR base and large, and AfroXLMR-Social (Adelani et al.). The tasks are AfriSenti three-way sentiment over 14 languages (Muhammad et al.) and MasakhaNER 2.0 named-entity recognition over 20 languages (Adelani et al.). Each runs at the full training size and at subsamples of 1,000 and 200 examples. We include the 200-example regime because Lai et al. show that this is where standard adaptation starts to overfit. That paper's evaluation is entirely in English, which leaves the human-language version of the regime open.

For each model, language, task and data size, we run a random search of up to 64 configurations. The search covers learning rate, warmup fraction, batch size, weight decay, epochs and dropout, with three seeds per configuration. For the large models, a stratified subset of languages gets the same treatment. We then compute the sensitivity metrics of Wiemerslage et al.. Rel@k is the best score found in k random configurations, as a fraction of the overall best. We also count how many configurations land within 1% of the best, and how many separate local maxima the search finds. These measurements extend that paper's method from character-level sequence models to pretrained encoders on sentence-level and token-level tasks.

The spectral probe works as follows. First we attach the task head and train only the head for a few hundred steps, so that the curvature is not dominated by random head weights. Then we take a fixed batch of target-language training data and use a Lanczos iteration with Hessian–vector products to estimate the top $r$ eigenvectors of the loss Hessian, the matrix of second derivatives of the loss with respect to the weights. Each eigenvector $v$ is a unit vector over all weights. We split it into parameter blocks $b$: the query, key, value and output projection and the feed-forward weights, separately for every layer. The mass on block $b$ is $m_b = \|v_b\|^2$. Localization is the effective number of blocks the curvature occupies:

$$ N_{\text{eff}}(v) = \Big(\sum_b m_b^2\Big)^{-1}, $$

averaged over the top $r$ eigenvectors and divided by the total number of blocks $B$. A value near $1/B$ means curvature sits in one block, and a value near $1$ means it is spread evenly.

Grebenkov and Nguyen warn that whether a mode counts as localized depends on which norm is used to measure it. We therefore also report a version based on the fraction of mass in the top blocks, and require that our conclusions hold under both.

We also test a cheaper variant that avoids solving for eigenvectors at all. It uses the block-wise trace of the Gauss–Newton approximation to the Hessian. The motivation comes from Lu et al., who predict where physical eigenfunctions localize from a smoothed potential without solving the eigenproblem. We test whether an analogous shortcut works for neural networks, and we expect it may be weaker.

Baselines

Naive default recipe. This is the reference that published leaderboards actually use. Every model is fine-tuned once per language with the fixed recipe from the original benchmark papers. We report the gap between this and the tuned score for every pair. As external anchors we use the numbers reported in Muhammad et al. for the top AfriSenti systems: weighted F1 of 71.31 on monolingual Task A, 75.06 on multilingual Task B, and 58.15 average F1 on zero-shot Task C. These are tuned shared-task systems, not single-recipe models. They tell us whether our tuned encoders fall in a credible range; they are not a quantity we expect to beat. For MasakhaNER 2.0, the file gives cross-lingual figures, an average 14 F1 advantage of African-source over English-source transfer and 85.5 F1 for isiXhosa to isiZulu. It gives no per-model in-language score, so we found no reported result on Lacuna for an in-language default-recipe baseline per model, and we will produce one.

Strongest model under fair tuning. The strongest competitor is the Africa-centric encoder family, AfroXLMR-large and AfroXLMR-Social, because existing evidence favors them on exactly these tasks. Every model gets an identical search space, number of configurations, number of seeds and GPU-hour cap. Selection happens on validation data only, and test sets are scored once at the end. For the empirical claim, we count a ranking as having flipped only if the tuned difference between two models changes sign and the bootstrap confidence intervals over seeds and test examples exclude zero in both directions.

Predictor baselines. The localization score has to beat three cheaper predictors. The first is the top Hessian eigenvalue alone, which is the sharpness of the loss. The second is a 10-point learning-rate range test, which is a short sweep of learning rates with a few hundred steps each. The third is a tokenizer statistic: the average number of subword pieces per word in the target language, which captures fragmentation. The localization score counts as useful only if adding it to these three improves Spearman correlation with measured Rel@8 on held-out languages, with a bootstrap interval that excludes zero.

Ablation of block resolution. To isolate the key ingredient, we recompute the same participation measure after randomly permuting weights across blocks of equal size. If the permuted score predicts as well as the real one, then the signal comes from overall curvature scale and not from how curvature is distributed across blocks, and the mechanistic claim fails.

Why not just…?

Why not just run random search for every model and language, and skip the spectral probe? For the empirical claim, that is exactly what we do. Exhaustive search is the ground truth of this study, not a competitor. The probe matters for what comes after. A 64-configuration, three-seed search over seven models, 34 language–task pairs and three data sizes costs far more than the communities that build these benchmarks can usually afford. The probe is useful only if it lets later benchmark builders spend their budget on the pairs that need it. If the budget-allocation experiment below shows no gain over uniform allocation, we will report the re-evaluation as the contribution and drop the claim that the probe saves compute.

Why not just use successive halving or Hyperband to cut the search cost? These methods train many configurations briefly and keep only the most promising ones for further training. This is a real threat to the probe's practical value. Successive halving spends compute adaptively without any theory, and it may match whatever savings the probe offers. We will include it directly in the allocation experiment at the same total budget. The probe does have two properties that successive halving lacks. It acts before any configuration has been trained, so it can decide how many configurations each pair receives, and the two methods can be combined. It also explains sensitivity in terms of where curvature sits. If Hyperband alone matches the probe-guided allocation, we will say so plainly and keep the probe only as an explanation.

Why not just measure sharpness, or count tokenizer fragmentation? Fragmentation is the most plausible single cause of sensitivity on agglutinative languages (Meyer et al.). Through the input-magnitude scaling of Ormaniec et al., fragmentation may also be the upstream cause of any curvature concentration we find. Both sharpness and fragmentation are therefore included as baselines. The held-out correlation test decides whether localization adds information beyond them. If it does not, we report that the simpler statistics are sufficient.

Why not just fine-tune with a gating method like JoLA, which already localizes which heads to edit? Lai et al. learn sparse gates over attention heads from 200 examples. They report 70.55% reasoning accuracy against 66.58% for LoRA on LLaMA-3-8B. That method answers a different question: which parameters to adapt, not whether a benchmark's comparison between models is fair. It also has temperature and sparsity-penalty hyperparameters that the paper never reports sensitivity for. As a secondary analysis, we check whether the blocks where curvature concentrates overlap with the heads JoLA keeps open when run on AfriSenti at 200 examples. We treat that overlap as a point of contact between the two lines of work, not as evidence for our core claims.

Proposed Experiments and Decisive Checks

The first decisive check is the flip rate. For every pair of models on every language, task and data size, we compare the ordering under the default recipe with the ordering under budget-matched tuning, and report the fraction of significant reversals. The empirical claim fails if fewer than one in twenty orderings reverse and the tuned-minus-default gains are statistically equal across model families. In that case the field's conclusion stands, and we will say so. We additionally test whether Africa-centric models still lead in tuned zero-shot transfer on AfriSenti Task C, where Azime et al. report Oromo scores as low as 42.09% F1. This is the regime where noisy validation sets most affect hyperparameter selection.

The second check is prediction on held-out languages. We fit the predictors on half the languages and evaluate Spearman correlation with Rel@8 and with the gap between default and tuned scores on the other half. We repeat this with splits grouped by language family, so that Nguni and Bantu languages are never in both halves. The mechanistic claim fails if localization adds nothing over sharpness, the learning-rate range test and fragmentation, or if the block-permuted ablation predicts equally well.

The third check is allocation. We give a fixed total budget of configurations for the whole benchmark to four strategies: uniform allocation, successive halving, probe-proportional allocation, and probe-proportional allocation combined with successive halving. Each strategy is scored by its average shortfall from the best result found by the full search, and by how many of the true orderings it recovers. Separately, we report how localization changes from 1,000 to 200 training examples. This measures whether data scarcity itself concentrates curvature, which is the link between scarce data and fragile defaults that the whole proposal rests on.

All configurations, seeds, curvature probes and scores are released together with the harness. The point is that a later Africa-centric model can be compared under the same rules instead of a new default recipe.