Arabic language technology underserves minority regional dialects. Before anyone can fix that, they need to know how much minority-dialect text a corpus actually contains, and today that number is mostly guessed. The usual approach labels each sentence with a classifier or a geolocation tag and counts the labels. Both are unreliable for the varieties that matter most: classifiers for rare dialects are trained on very little data, and geolocation proxies were only 49.3% accurate for Tunisia in NADI 2024. Particle physicists face the same problem when they count a rare signal buried in background. Their answer is not to classify each event and count. They fit the whole sample at once with an unbinned extended maximum likelihood fit: a statistical fit that models every event's continuous measurements as drawn from a mixture of "signal" and "background" distributions, and treats the total number of events as a Poisson count. The fit returns the signal yield directly, together with an honest error bar.
We propose to transplant that method to dialect auditing. Each sentence is embedded with an Arabic encoder. Principal component analysis (PCA, which finds the few directions along which the data varies most) compresses the embedding to a handful of coordinates, so that a properly normalized multivariate density becomes tractable. An extended likelihood fit over those coordinates, plus a continuous dialectness score, then estimates how many sentences of a target dialect the corpus contains, with confidence intervals and upper limits. The claim is narrow and testable. When only tens of labeled sentences of a minority dialect exist, a fit that lets the dialect's distribution shape float in a low-dimensional PCA space should estimate prevalence with less bias and better-calibrated uncertainty than counting classifier outputs, including bias-corrected counting.
Core Claim
The quantity we estimate is a corpus-level yield, not a per-sentence label. In physics, rare-signal analyses such as the CLEO II observation of B+ → ωK+ fit a joint density over a few kinematic variables and an event-shape discriminant. The τ → μγ search used the same kind of fit and reports outperforming box-counting cuts, and sets an upper limit when no signal is seen. The BASiL on/off method goes further. It replaces hard selection cuts with event-by-event likelihoods and uses a signal-free "off" region to pin down the background, which gave a 15–20% precision gain at 10% signal-to-noise. These papers supply our method. Dialect NLP supplies the motivation. NADI 2024 found that 70% of sentences were valid in two or more dialects, so hard per-sentence labels are partly ill-posed. ALDi showed dialectness is a continuous quantity with a multimodal distribution. Both findings favor fitting distributions over counting labels.
Written out, for a corpus of $N$ sentences with compressed coordinates $\mathbf{z}_i$ (top PCA components plus an ALDi score), the fit maximizes
$$\mathcal{L}(\nu_S,\nu_B,\boldsymbol{\theta}) = e^{-(\nu_S+\nu_B)} \prod_{i=1}^{N}\left[\nu_S\,P_S(\mathbf{z}_i\mid\boldsymbol{\theta}_S) + \nu_B\,P_B(\mathbf{z}_i\mid\boldsymbol{\theta}_B)\right] \cdot \prod_{j=1}^{m} P_S(\mathbf{z}^{\text{lab}}_j\mid\boldsymbol{\theta}_S)\, \cdot \prod_{k} P_B(\mathbf{z}^{\text{off}}_k\mid\boldsymbol{\theta}_B).$$
Here $\nu_S$ and $\nu_B$ are the expected counts of target-dialect and other sentences. $P_S$ and $P_B$ are their densities, with parameters $\boldsymbol{\theta}_S$ and $\boldsymbol{\theta}_B$. The $m$ vectors $\mathbf{z}^{\text{lab}}_j$ are the few labeled target-dialect sentences, and the $\mathbf{z}^{\text{off}}_k$ come from an "off" reference corpus known to lack the target dialect, such as MSA news plus dominant dialects. The labeled examples only constrain the signal shape. The unlabeled corpus itself also informs that shape, which is what lets the method work when $m$ is tiny. The 68% and 95% intervals on $\nu_S$ come from the profile likelihood.
Three pieces of prior work bound the novelty. The link between likelihood fits and PCA in maximum likelihood i-vector space using PCA does per-utterance speaker scoring, with no yield estimation. Geometric constrained MLLR adapts acoustic models to Mandarin dialects but never measures how much of a dialect is present. Dialect separation via log-likelihood score distributions uses likelihoods only as a post-hoc separability diagnostic. None of the three estimates corpus prevalence with uncertainty.
Research Program
The pipeline has four parts.
Encoder and dialectness coordinate. We use a MARBERT sentence encoder, as in ALDi. An ALDi regressor (RMSE 0.18 on AOC-ALDi per ALDi, 0.1403 on NADI 2024 per ASOS) supplies the dialectness coordinate. It plays the role of the event-shape discriminant in the CLEO fit.
PCA compression. PCA reduces the embedding to $K \in \{4, 8, 16\}$ dimensions, where Gaussian-mixture or small normalizing-flow densities normalize exactly. This sidesteps the high-dimensional integration problem that confines physics fits to a few variables. Choosing which corpus the PCA basis is fitted on is itself a research question. Ma et al. found that PCA retains about 90% of variance in-domain but degrades sharply out of domain. A basis fitted on MSA-heavy text may therefore drop exactly the directions that carry minority-dialect signal. We compare three bases: one fitted on MSA, one on the pooled corpus, and one on the audited corpus itself. Each is tested with and without the dominant-direction removal of Raunak et al.. That removal step subtracts the mean and the few highest-variance directions, which are typically uninformative. In our setting those directions may encode MSA-versus-dialect register, so removing them could help or hurt.
Density models. The background density is a flexible mixture fitted jointly on the "off" corpus. The signal density is a low-capacity family, either a Gaussian or a two-component mixture to reflect ALDi's multimodality. Its shape parameters float in the fit, constrained by the $m$ labeled examples.
Statistical machinery. We add profile-likelihood intervals and coverage checks on pseudo-experiments, a standard physics practice. We also add upper limits for the case where the target dialect may be absent, mirroring the τ → μγ limit-setting.
Baselines
The naive reference point is metadata counting. It assigns each tweet the dialect of its geolocated country and counts. This is how many corpora are described in practice, and NADI 2024's 49.3% Tunisian geolocation accuracy suggests it is badly biased. We run it wherever geolocation exists. A second naive baseline is classify-and-count. A MARBERT classifier is fine-tuned on all available labels, including the same $m$ target examples our method sees, and its argmax predictions are counted. For a sense of classifier quality, the best reported per-sentence results are 67.32% macro F1 on MADAR 25-city identification and 71.70% on MADAR Twitter country identification (MADAR shared task), and 50.57 macro F1 on NADI 2024 multi-label identification. These per-sentence results bound how good counting can be. However, no reported result was found on Lacuna for corpus-level dialect prevalence estimation on any of these datasets, so the yield baselines must be run by us.
The strongest existing competitors are standard quantification corrections applied to that same classifier, which we implement ourselves since the file cites no paper for them. Adjusted classify-and-count inverts the classifier's confusion rates, measured on held-out labeled data. Expectation-maximization prior re-estimation iteratively re-weights the calibrated classifier posteriors to match the corpus. The latter is close kin to a mixture fit, and it is the baseline we most need to beat. The sharpest competitor is a one-dimensional unbinned extended fit on the classifier's score alone. This mirrors physics analyses that fit only a Fisher discriminant. It isolates whether the multi-dimensional PCA coordinates and the floating signal shape add anything beyond a good score.
For fairness, every method gets the same encoder checkpoint, the same $m$ labeled target sentences, the same off-corpus, and the same hyperparameter search budget. Classifier-based methods additionally get a held-out calibration split drawn from the same labels. Ablations remove one ingredient at a time. We fix the signal shape to the labeled examples instead of floating it. We drop the ALDi coordinate. We swap the PCA basis source. We replace the extended term with a fixed-$N$ likelihood, which should leave point estimates roughly unchanged but alter uncertainty when yields are compared across corpus slices. Beating the field means two things across the low-label regime ($m \le 50$) and low prevalence (1–5%). First, lower absolute yield error than both expectation-maximization re-estimation and the one-dimensional score fit. Second, 95% intervals whose empirical coverage is within a few points of nominal while the baselines' coverage falls short.
Why not just…?
Why not just count classifier predictions and correct for confusion? Adjusted counting is unbiased only if the classifier's error rates measured on labeled data carry over to the audited corpus. With $m$ in the tens, those rates are themselves very noisy, and NADI's overlap finding means some "errors" are not errors at all. Still, this is a legitimate competitor. If it matches our fit when $m$ is small, the extra machinery is not justified, and the label-budget sweep will show which case holds.
Why not just run EM prior re-estimation on classifier posteriors? This is the most serious threat, because it already is a mixture likelihood, just over posteriors rather than over coordinates. Its weakness is that it trusts the classifier's calibrated posterior shapes, which come from the few labels. Our fit lets the unlabeled corpus reshape the signal density. If EM matches us, the honest conclusion is that the physics framing adds only intervals and upper limits. Even that is worth reporting, because EM gives no profile-likelihood coverage or limit-setting out of the box.
Why not just fit the unbinned likelihood on the one-dimensional classifier score? Physics often does exactly this. It may work as well as our method once labels are plentiful. Our hypothesis is that a score trained on tens of examples discards structure that the PCA coordinates retain. The direct test is our method versus the score-only fit at matched $m$. If the score-only fit ties everywhere, we will report that PCA is unnecessary and simplify the method to the score-only fit.
Why not just prompt a large model to label each sentence and count? This is classify-and-count with a different classifier. It inherits the bias problem and still gives no uncertainty. The global linguistic hierarchy analysis argues these models represent low-resource varieties worst, so their errors should concentrate on the target dialects. We include one prompted labeler as an additional classify-and-count arm at matched inference cost.
Proposed Experiments
Controlled mixtures on MADAR Travel. The decisive experiment uses the MADAR Travel corpus, whose parallel sentences across 25 cities and MSA hold content fixed. This means a yield estimate cannot succeed by detecting topic. We build thousands of synthetic corpora with a known fraction (0, 0.5%, 1%, 2%, 5%, 10%) of a held-out target city, chosen to include Maghrebi cities such as Rabat and Tunis. Background is drawn from MSA and dominant dialects. We sweep $m$ from 5 to 500 labeled target sentences. Outcomes are absolute and relative yield error, interval coverage, and false-positive rate of the upper limit in zero-signal corpora.
Natural data. We repeat the experiment on MADAR Twitter, with country as the target, and compare against the geolocation-count baseline.
Overlap check. On NADI 2024 multi-label data, the target is the number of tweets annotated as valid in a given country's dialect. Here the per-sentence ground truth is genuinely non-exclusive, so this tests whether distribution fitting handles overlap better than counting labels.
PCA basis shift. We measure how the basis source changes yield bias for Maghrebi targets. This directly tests whether the out-of-domain PCA degradation reported by Ma et al. reappears under dialect shift, and whether a basis fitted on the audited corpus repairs it.
Falsification criteria. The proposal fails if, at $m \le 50$ and prevalence $\le 5\%$, the multi-dimensional fit does not reduce yield error relative to EM re-estimation and the score-only fit, or if its intervals under-cover on the controlled mixtures.
References
- NADI 2024: The Fifth Nuanced Arabic Dialect Identification Shared Task
- Observation of<mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:mrow><mml:msup><mml:mrow><mml:mi mathvariant="italic">B</mml:mi></mml:mrow><mml:mrow><mml:mo>+</mml:mo></mml:mrow></mml:msup></mml:mrow><mml:mo>→</mml:mo><mml:mrow><mml:msup><mml:mrow><mml:mi mathvariant="italic">ω</mml:mi><mml:mi mathvariant="italic">K</mml:mi></mml:mrow><mml:mrow><mml:mo>+</mml:mo></mml:mrow></mml:msup></mml:mrow></mml:math>and Search for Related<mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:mi mathvariant="italic">B</mml:mi></mml:math>Decays Modes
- Update of the search for the neutrinoless decay<mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:mrow><mml:mrow><mml:mover><mml:mrow><mml:mi>τ</mml:mi></mml:mrow><mml:mrow><mml:mo>→</mml:mo></mml:mrow></mml:mover></mml:mrow></mml:mrow><mml:mi>μ</mml:mi><mml:mi>γ</mml:mi></mml:math>
- Signal estimation in on/off measurements including event-by-event variables
- ALDi: Quantifying the Arabic Level of Dialectness of Text
- Maximum likelihood i-vector space using PCA for speaker verification
- Geometric constrained maximum likelihood linear regression on Mandarin dialect adaptation
- Dialect separation assessment using log-likelihood score distributions
- ASOS at NADI 2024 shared task: Bridging Dialectness Estimation and MSA Machine Translation for Arabic Language Enhancement
- Simple and Effective Unsupervised Redundancy Elimination to Compress Dense Vectors for Passage Retrieval
- Effective Dimensionality Reduction for Word Embeddings
- The MADAR Shared Task on Arabic Fine-Grained Dialect Identification
- Artificial intelligence is creating a new global linguistic hierarchy