Explore Then Refine: Recovering Clean Accuracy from Unnatural Augmentation in Low-Resource Languages

When a language has only a few hundred training sentences, practitioners pad the training set with synthetic variants: shuffled word orders, inserted noise, dictionary swaps, copied sentences. A consistent finding is that these variants help less, or actively hurt, when they look unlike what native speakers produce. Grammatically valid but rare Arapaho word orders degraded translation (Is linguistically-motivated data augmentation worth it?). Across code-switching augmentation methods, human-rated naturalness correlated 0.97 with translation quality (Data Augmentation Techniques for Machine Translation of Code-Switched Texts: A Comparative Study). The field has read this as a property of the data: unnatural synthetic text is bad text, and the fix is better generators or hundreds of hours of grammar engineering.

We propose a different reading, borrowed from work on why robust training lowers clean accuracy. In vision, heavy augmentation lowers the error gap between training and test data but raises error on clean training examples. It does this by distorting the model's internal features, and a short final phase of training on clean or lightly augmented data at low learning rate repairs the distortion (Data Augmentation Revisited: Rethinking the Distribution Gap between Clean and Augmented Data). If the same mechanism operates in low-resource text, then much of the harm from unnatural augmentation is a training-schedule artifact rather than an intrinsic flaw. Unnatural variants would then be useful early, as exploration, and naturalness would matter only at the end. This note proposes to test that claim directly on Indigenous-language translation, glossing and sentence transformation.

Core Claim

The claim is that the damage unnatural augmentation does to low-resource sequence models is mostly recoverable by a two-phase schedule. In the exploration phase the model trains on augmented data. In the refinement phase it trains on the original sentences only, at a reduced learning rate, for the last fraction of training. Two predictions follow and could be wrong. First, augmentations that hurt under ordinary single-phase training, such as Arapaho PERM and dictionary substitution, should become neutral or helpful under explore-then-refine. Second, the strong link between naturalness and downstream quality reported by Hamed et al. should weaken substantially under the two-phase schedule. That would mean naturalness governs what the model must fit at the end of training, not what it can usefully see along the way.

The robustness side of the trade-off is generalization to shifted test conditions. The gap between development and test performance on Yukatek Maya sentence transformation came from rarer transformations and unseen stems (On the Robustness of Neural Models for Full Sentence Transformation). Augmentation is supposed to buy robustness to exactly this kind of shift. The question is whether refinement can recover in-distribution accuracy without giving that robustness back. If the shifted-test gains vanish along with the clean-accuracy losses, refinement has simply undone the augmentation, and the claim fails.

Relation to Prior Work

Three papers motivate the proposal by documenting the harm. Groshan et al. compare naive and linguistically engineered augmentation for Uspanteko and Arapaho with ByT5-small and find that grammaticality does not guarantee utility. Hamed et al. tie naturalness to translation quality for Egyptian Arabic–English. The full-sentence-transformation study finds that combining augmentations gives diminishing or negative returns and leaves the distribution gap without a remedy. All three train in a single phase, so none can separate "the data is bad" from "the model ends training fit to the wrong distribution."

He et al. supply the method and the diagnostic: risk decomposition and singular-value analysis of penultimate features. They test only continuous Mixup-style augmentation on images, and the transfer to discrete, morphologically complex text is the open question. TART (TART: Boosting Clean Accuracy Through Tangent Direction Guided Adversarial Training) and manifold purification (Breaking the Adversarial Robustness-Performance Trade-off in Text Classification via Manifold Purification) attack the same trade-off by modeling the clean data's geometry. They act as a constraint on our design: both need dense data to estimate that geometry, which a 100-sentence corpus cannot provide. We therefore deliberately avoid any learned model of the data manifold and change only the schedule.

The closest connecting work is decision-boundary-aware augmentation (Enhancing Effectiveness and Robustness in a Low-Resource Regime via Decision-Boundary-aware Data Augmentation). It improves clean and adversarial accuracy on subsampled English classification by generating ambiguous examples. It changes what is generated, and its authors leave the covariate shift of the generated data unresolved. We leave generation untouched and target that shift through training dynamics, on generation tasks in genuinely low-resource languages.

Research Program

We will reproduce the Groshan et al. setting: ByT5-small, Uspanteko and Arapaho translation and interlinear glossing, training sizes of 100, 500, 1,000 and 5,000. We use their naive augmentations (INS-NOISE, DEL-ANY, DUP) and their engineered ones (UPD-TAM, INS-CONJ/INTJ, PERM). Each augmentation is trained single-phase and explore-then-refine. The refinement fraction and the refinement learning rate are tuned on development data.

On the AmericasNLP 2024 sentence-transformation data for Bribri, Yukatek Maya and Guaraní (Findings of the AmericasNLP 2024 Shared Task on the Creation of Educational Materials for Indigenous Languages), we repeat the comparison with COPY, COPYext, TRANS, PER and CON. Each language is evaluated separately on the development split and the shifted test split.

For the naturalness test we take the Hamed et al. methods, whose method-level naturalness rates are reported: back-translation was rated natural 80% of the time and LEXDict 16%. We subsample the ArzEn-ST parallel data to low-resource sizes. Under each schedule we then measure how much of the chrF++ gap between the most and least natural methods survives.

Alongside task scores we record two quantities. The first is clean training loss, which serves as the empirical-risk term. The second is the singular-value spectrum of encoder and final decoder hidden states on clean inputs. Together they show whether the He et al. mechanism (feature distortion that refinement repairs) actually operates in byte-level seq2seq models, or whether any gains come from something else.

Baselines

The naive reference point is ByT5-small trained on the original sentences alone, with the same total number of updates as every augmented run.

The strongest existing method is the best single-phase augmentation for each language and size, as chosen by Groshan et al. They report gains of up to +8 chrF on Uspanteko and up to +3 chrF on Arapaho at 100 examples (Is linguistically-motivated data augmentation worth it?). These are reported gains over no augmentation, not absolute scores, so we will re-run their setup to obtain absolutes. Beating this baseline means a two-phase model, using either the same augmentation or a cheap naive one, exceeds that best single-phase chrF. The win must hold across five seeds with non-overlapping bootstrap intervals, at 100 or 500 examples.

For sentence transformation there is no reported result on Lacuna for per-system test accuracy or chrF in the file. Our baseline there is the best single-phase COPY-family configuration, re-run by us. For code-switched translation, the reference numbers are 58.6 chrF++ for back-translation in the non-zero-shot setting and about 56.0 for EC and LEXRand zero-shot (Data Augmentation Techniques for Machine Translation of Code-Switched Texts: A Comparative Study). Those numbers come from the full corpus, so they bound our subsampled runs rather than serving as direct targets.

Two ablations isolate the key ingredient. The first, refine-without-explore, trains on clean data for the full budget with the same learning-rate drop, which isolates the contribution of the augmented phase. The second refines on lightly augmented data instead of pure clean data, which tests whether the final phase needs the exact natural distribution. Every method gets the same total steps, the same 20-trial tuning budget over its own hyperparameters, and the same development split for selection.

Why not just…?

Why not just tune the mixing ratio of augmented to real data? This is the most serious threat. A well-chosen static down-weighting of synthetic examples might reach the same endpoint without any schedule. We will tune the ratio with the same 20-trial budget and compare. If static mixing matches explore-then-refine on both clean and shifted-test scores, the scheduling claim is wrong and we will report it that way. The naturalness analysis would still stand, because it asks how much harm is recoverable at all, not which recovery tool works best.

Why not just filter augmented sentences for naturalness? Filtering presumes a reliable naturalness scorer. For Arapaho or Bribri no fluent language model exists, and perplexity-based scoring behaves poorly on low-resource text: even on English, perplexity filters reject about 10% of benign inputs (Baseline Defenses for Adversarial Attacks Against Aligned Language Models). Filtering also discards exactly the exploration signal we hypothesize is useful. We will still include an oracle-style filter on ArzEn-ST, where method-level naturalness is known, as a comparison.

Why not just select the best checkpoint on development data? Early stopping picks a point on a single trajectory. It cannot undo distortion that accumulates throughout augmented training, and selection on development data is precisely what failed to predict the Yukatek test drop. Every one of our runs already uses development-based checkpoint selection, so any remaining gain from refinement is beyond what selection alone provides.

Why not just invest in linguistically engineered augmentation? Groshan et al. report over 200 hours of grammar formalization with gains that are not proportional to that effort. If refinement makes INS-NOISE or DUP match UPD-TAM, that becomes a direct cost argument. If engineered augmentation still wins after refinement, the expert labor is vindicated.

Decisive Checks

The proposal rests on three results. First, PERM on Arapaho must move from harmful to neutral or helpful under explore-then-refine at 100 and 500 examples, and clean training loss and the feature spectrum must move back toward the no-augmentation model. Second, the chrF++ gap between back-translation and LEXDict on subsampled ArzEn-ST must shrink by at least half under refinement; if it does not, naturalness is a property of the data after all. Third, on Yukatek Maya, the gains on the shifted test split must survive refinement while development accuracy recovers. If refinement erases them, the trade-off is real in this regime and not an artifact of the schedule.

We expect all effects to fade toward 5,000 examples, consistent with the diminishing augmentation gains Groshan et al. observed. Seeing that fade would itself be evidence that the mechanism is the data-scarcity distortion He et al. describe.