Audit-Driven Filter Selection for Script-Polluted Low-Resource Web Text

Web-crawled text for rare languages is often not in those languages. Audits find whole subsets with no usable text. In mC4, only about 57% of low-resource-language text is in the correct writing system, compared with 92% for the corpus overall. Today's cleaning pipelines respond with one fixed recipe per language: a language identifier, a script check, and a threshold. A recipe strict enough to stop pollution also throws away real dialectal text, and a recipe lenient enough to keep that text lets English boilerplate and wrong-script strings through. This proposal treats the choice of cleaning recipe as an online decision made separately for each source website. A small budget of human audits tells the system, website by website, whether it can afford to be lenient.

The idea is enabled by a feature of filtering that ordinary algorithm-selection problems lack. Every candidate filter can be run on every document at negligible cost. Once one document is audited, we therefore know how every recipe would have fared on it, not just the recipe we used. This gives us full counterfactual feedback on whichever documents get audited. The only scarce resource is the audit itself, so the method audits only documents on which the recipes disagree. On top of this sits a "safe-by-default" controller borrowed from online learning. It starts from today's strict pipeline and moves to more lenient recipes only when audits show a website is clean. Its guarantee is that it never ends up more than a constant amount worse than the strict pipeline.

Core Claim

Script pollution is concentrated by source, not only by language. Lenient filtering is a safe bet on some websites and a bad one on others. Lignos et al. show that pollution comes from specific harvesting mechanisms, such as empty native fields falling back to English strings (Toward More Meaningful Resources for Lower-resourced Languages). CCNet and the OSCAR document-level work show it also comes from specific page templates (CCNet; Towards a Cleaner Document-Oriented Multilingual Crawled Corpus).

If pollution really varies this way, a controller that picks a recipe per website should beat any single per-language recipe. The comparison is made at a fixed contamination level, measures how much genuine text is kept, and spends only a few hundred audited documents per language. The claim is falsifiable at its root. If websites within a language differ little in which recipe is best, the per-language static tuning of FineWeb2 (FineWeb2) already captures the available gain. The first experiment below tests exactly this.

What Exists and What Is Missing

The curation literature supplies the candidate recipes and motivates the gap. GlotScript identifies the script of each character and drops sentences whose script does not match the language's known writing systems (GlotScript). GlotLID adds a language identifier for 1,665 languages with a fixed rejection threshold of 0.5 (GlotLID). OpenLID is an audited fastText identifier with class rebalancing (An Open Dataset and Model for Language Identification). MADLAD-400 uses document-level majority voting plus hand-written boilerplate regexes (MADLAD-400). FineWeb2 derives per-language thresholds offline from reference statistics. All of these fix the recipe before seeing the stream. Caswell et al. document the recall–precision tension that results but propose no mechanism to manage it (Quality at a Glance). We reuse these tools unchanged as the "arms," meaning the candidate recipes the controller chooses among. Our contribution is only in how they are chosen.

The algorithm-selection literature supplies the controller, and two of its results constrain the design. (A,B)-PROD combines a safe benchmark algorithm with an adaptive one. It guarantees constant regret against the benchmark (the controller loses at most a constant amount relative to it) while keeping $O(\sqrt{T})$ worst-case regret (Exploiting easy data in online optimization). That paper notes its easy-data guarantees need more than one-point bandit feedback, where only the chosen option's outcome is observed. Our setting escapes that limitation, because an audit reveals the outcome of every recipe. Best-of-both-worlds detect-and-switch methods start robust and switch to aggressive behaviour once a test statistic certifies a benign environment (Lee et al.). Amir et al. use an internal behavioural count as a free regime detector (Better Best of Both Worlds Bounds for Bandits with Switching Costs). We use the free disagreement rate between the script check and the language identifier in the same spirit.

Stream meta-learning maps cheap window statistics to the expected best classifier (Towards Meta-learning over Data Streams). We reuse that idea to share information across websites that have few audits. Degroote finds that exploration often hurts relative to pure greedy selection in online algorithm selection (Online Algorithm Selection). This finding constrains us: we include a greedy ablation and expect it to be competitive. TAILOR shows that when the natural reward saturates, a designed reward is needed (Algorithm Selection for Deep Active Learning with Imbalanced Datasets). In our setting the natural reward is unobserved rather than saturated, which is why the audit reward below is defined explicitly. None of these selection methods has been applied to text ingestion.

Research Program

The unit of decision is a stream $h$: one language on one web host, across Common Crawl snapshots. The candidate recipes are a small fixed portfolio of about six, kept small because regret grows with the number of arms, as Zhang et al. note. The portfolio runs from strict to lenient:

  • GlotScript mismatch rule, then document-level GlotLID at $\theta=0.5$, then MADLAD-style majority voting. This is the strict end.
  • The same, with line-level boilerplate stripping applied before identification.
  • Macrolanguage-level identification, so that related "cousin" varieties are not discarded.
  • Lowered rejection thresholds.
  • Script check only. This is the lenient end.

For a recipe $\pi$ on stream $h$, we define the utility of the text it keeps as

$$u_h(\pi) = C_h(\pi) - \lambda\, P_h(\pi),$$

where $C_h(\pi)$ is the number of correct-language tokens the recipe keeps and $P_h(\pi)$ is the number of kept tokens that auditors label as wrong language, wrong script, or non-linguistic, following Caswell et al.'s categories. The parameter $\lambda$ sets how many good tokens one polluted token is worth. We sweep $\lambda$ to trace a full recall–contamination curve rather than report a single operating point.

Auditing follows from the full-feedback observation. Documents on which all recipes agree carry no information about which recipe is better, so the auditor draws only from the disagreement set. Sampling within that set is proportional to how much the recipes' decisions differ in tokens. Audits are importance-weighted so that the estimates of $u_h$ remain unbiased.

The controller follows (A,B)-PROD. The benchmark B is the strict recipe currently used in production, and A is a Hedge learner over the whole portfolio. This gives a deployment-relevant guarantee: on estimated utility, the controller is never more than a constant worse than the existing pipeline. It also converges toward the best recipe for each host. The guarantee holds for the audit-estimated utility, not the true one, so audit noise enters directly. We will report how the estimation error scales with audits per host.

Hosts with few audits borrow strength from a contextual model, as in van Rijn et al. That model predicts the best recipe from free per-host features: the script-versus-identifier disagreement rate, the share of Latin script within a non-Latin language, the share of repeated lines across pages, and the identifier's confidence distribution. The disagreement rate also serves as an unaudited regime-change detector. A jump in it re-opens auditing on a host whose template has changed.

Baselines

The naive reference point is the corpus as harvested, with only the source's own language tag. For mC4 low-resource subsets, GlotScript reports average script accuracy of 0.566, the floor against which contamination reduction is measured.

The strongest existing method is the FineWeb2 pipeline, which combines GlotLID, deduplication, and per-language thresholds fitted offline. It is the strongest competitor because it already adapts per language, which is the obvious coarser version of our idea. Its reported evidence is that its data yields "more performant models than prior multilingual datasets on 11 out of 14 languages." We found no reported result on Lacuna for document-level precision or recall of FineWeb2's filtering on audited low-resource web text, so we will measure it ourselves. We run it on the identical snapshot sample. It also receives an extra advantage: the same audit budget, used to re-fit its per-language thresholds offline. Beating it means keeping significantly more correct-language tokens at equal or lower audited contamination. Significance is assessed by a bootstrap over hosts, for the majority of evaluated languages.

The best static language identifier is the second strong reference. On language identification itself, OpenLID reports macro-average F1 of 0.93 and a false positive rate of 0.033 on FLORES-200*. GlotLID reports more than 12 points of absolute F1 over its baselines on UDHR. We use GlotLID with the GlotScript pre-filter as the strict benchmark arm, so any gain must come from selection rather than a better identifier.

Three ablations isolate the key ingredients. The first draws audits uniformly instead of from the disagreement set. The second applies one recipe per language instead of per host. The third makes greedy choices without the (A,B)-PROD safety layer, which Degroote's results suggest may be competitive.

Why not just…?

Why not just tune per-language thresholds offline, as FineWeb2 does? This is the main threat. It wins whenever the best recipe is roughly the same across hosts within a language. Experiment one measures how much of the variance in the best recipe lies between hosts within a language, compared with between languages. If the between-host share is small, we drop per-host selection and report that per-language static tuning is sufficient. The oracle-per-host versus oracle-per-language gap bounds what this proposal can possibly gain before any controller is built.

Why not spend the audit budget on fine-tuning the language identifier or training a quality classifier? This is a plausible alternative, since audited documents are training labels. A few hundred labels per language, though, are unlikely to fix the "cousin" collapse onto high-resource sink languages that GlotLID documents, or the Arabic dialect F1 of 0.002 that OpenLID reports. Those failures stem from scarce training data, not from a miscalibrated threshold. We test this directly. With the same audits, we fine-tune GlotLID on the labelled documents and compare it with selection. We also run the combination, because the two approaches are not exclusive.

Why not run every filter and keep only what they all accept? Requiring every filter to accept a document is the strict end of our portfolio. Its known cost is the recall loss Caswell et al. describe for colloquial and dialectal text. It appears as a baseline at every value of $\lambda$, so the curves show directly whether being lenient on clean hosts is worth anything.

Proposed Experiments

The evaluation set is built from low-resource-language subsets of mC4, OSCAR, and FineWeb2, grouped by host. We target about 40 languages, chosen to include non-Latin scripts with known fallback problems (Ge'ez, Syllabics, Sinhala), closely related language pairs, and Tetun, which has an audited corpus, Labadain-30k+. Native-speaker auditors label a held-out stratified sample using Caswell et al.'s categories. The controller's own audits are simulated by revealing held-out labels, which lets us sweep audit budgets cheaply.

The first experiment is the variance decomposition described above, which decides whether the project continues. The main experiment plots correct tokens retained against audited contamination for every method at budgets of 50, 200, and 1,000 audits per language. It also reports cumulative regret against the best recipe for each host in hindsight.

As a second stream with a different pollution mechanism, we apply the method to WikiAnn silver named-entity data. We then repeat Lignos et al.'s MasakhaNER experiment with XLM-R, where adding unfiltered silver data lowered F1 by 0.49 on average. The decisive check is whether silver data filtered by our controller turns that drop into a gain, and whether data filtered by the static pipeline does not.

Downstream, we train small language models on each filtered corpus at matched token counts. They are evaluated on SIB-200, Belebele, and the FineWeb2 canary tasks. We will report whether the gain in retained tokens survives once corpus size is held fixed, or comes only from having more text.