Learning Which Informal Spellings to Keep: Trust-Region Search over Normalization Policies

Social-media text is full of non-standard writing: "2moro," "ppl," "yaaaaaay," casual sentence endings, and kana and kanji variants. The usual fix is lexical normalization, which rewrites every such token into its dictionary form before a downstream model sees it. That fix is known to be lossy. Normalizing some kinds of variation helps a sentiment classifier, while normalizing others, such as pronunciation variants and vowel elongations, removes the emotional signal the classifier needs. Today the choice of what to normalize is made by hand, one category at a time, for one task and one language. This proposal treats that choice as a black-box optimization problem. Each candidate is a normalization policy, meaning a yes/no decision for each of dozens of edit categories. Scoring a policy requires retraining the downstream model on text normalized under that policy, which makes every evaluation expensive. The search is run with local surrogate modeling, a family of methods that fits a cheap statistical model of the objective only inside a small, adaptively sized neighborhood of the best policy found so far.

Stated in two sentences: given text annotated with typed normalization edits, we search for the subset of edit types whose application maximizes a downstream task metric, using trust-region Bayesian optimization because each evaluation costs a fine-tuning run and the space has about 2^33 points. The output is a better task score and also a measured map of which kinds of informal writing carry task-relevant signal and which are noise. That question the field has so far answered only by one-at-a-time ablation.

Core Claim

The tension is documented, but nobody has optimized against it. Text Normalization for Sentiment Analysis in Japanese Social Media annotated 6,000 Japanese posts from the WRIME corpus with a taxonomy of 6 classes and 33 subcategories of edits. With full manual normalization, BERT-large sentiment agreement (quadratic weighted kappa, QWK) rose from 0.511 to 0.589. Several individual categories gave gains of more than 3 QWK points, while normalizing pronunciation variants alone made performance worse. Adapting Sequence to Sequence Models for Text Normalization in Social Media makes the same point qualitatively for elongations in English. Both papers motivate this proposal: they show that "normalize everything" is not the optimum. Neither searches the space of partial policies, and per-category ablation cannot detect interactions, for example a case where normalizing abbreviations helps only when sentence endings are also normalized.

Our claim is twofold. First, a trust-region search over subsets of edit categories finds policies that beat full normalization by a margin larger than seed noise, within a budget of roughly 150 fine-tuning runs. Second, the resulting policies are task-dependent. Edits that are noise for dependency parsing are signal for sentiment, so no single hand-chosen policy transfers across tasks.

The choice of local over global surrogates is deliberate. Understanding High-Dimensional Bayesian Optimization shows that global surrogates in spaces of this size produce nearly flat acquisition surfaces, which give the search no useful direction. Recent Advances in Bayesian Optimization summarizes the trust-region premise that a rough function looks simple in a small neighborhood. Both supply the method's rationale.

Research Program

We first build the search space. On WRIME, every gold edit carries one of the 33 subcategory labels. A policy is therefore a 33-bit vector, and applying it means executing exactly the gold edits whose categories are switched on. This oracle setting isolates the question "which kinds of normalization help?" from normalizer errors. A second, realistic setting replaces gold edits with the outputs of a fine-tuned BART normalizer, the system Kondo et al. report gives moderate gains. Each predicted edit is assigned a category by a small classifier trained on the gold annotations.

To test transfer to another language and task, we use MultiLexNorm, which has no category labels. There we derive a coarser automatic edit typology from source–target pairs, with types such as elongation collapse, case change, split or merge, abbreviation expansion, and phonetic or orthographic substitution. Policies over these types are scored by downstream dependency parsing and part-of-speech tagging.

The objective $f(z)$ for a policy $z \in \{0,1\}^{d}$ is the validation metric of a downstream model fine-tuned and evaluated on text normalized under $z$. The metric is QWK for WRIME and labeled attachment score for MultiLexNorm. Because fine-tuning is noisy across random seeds, the surrogate is a Gaussian process (a probabilistic regression model that also reports its uncertainty). It uses a Hamming-distance kernel over policies and an explicit observation-noise term.

Following BOiLS, which defines discrete trust regions as Hamming balls around the incumbent sequence, the search restricts candidate policies to bit vectors within Hamming radius $r$ of the current best policy. The radius grows after consecutive improvements and shrinks after consecutive failures, as in TuRBO-style methods. BOiLS searches ordered sequences over an 11-symbol alphabet using a sub-sequence string kernel. Our space is an unordered set of edit types, so we use the simpler kernel and reuse only its trust-region mechanics.

Two safeguards address a known failure mode. LOL-BO showed that strong local optimizers exploit flaws in proxy objectives; in that work they produced chemically absurd molecules. Our analogue is overfitting the 500-post WRIME validation set. We therefore re-evaluate the final incumbent with five seeds on the untouched 500-post evaluation split. We also report the gap between validation and test as a first-class result.

The analysis component reads off the learned structure. We fit a second-order model of the main effects and pairwise interactions of category toggles to all evaluated policies. This tells us whether the optimum is close to additive, in which case simple methods suffice, or whether it involves interactions.

Baselines

The naive reference point is no normalization: BERT-large fine-tuned on raw WRIME text, for which Kondo et al. report a QWK of 0.511. The strongest existing configuration is full manual normalization, for which Text Normalization for Sentiment Analysis in Japanese Social Media reports a BERT-large QWK of 0.589. Beating it means a test QWK above 0.589 whose 5-seed confidence interval excludes our own re-run of full normalization. We re-run that baseline under identical seeds rather than trusting the published figure.

The strongest informed heuristic follows directly from the same paper: normalize everything except pronunciation variants. It costs a single evaluation. A search method that cannot beat it is not worth its budget.

On MultiLexNorm, the reference normalizer is the winning ÚFAL-2 ByT5 system. It reports a macro-averaged error reduction rate of 67.3% against MoNoise's 49.0%, and parsing gains of up to 1.72 labeled-attachment points. We apply its outputs in full and compare against our selectively applied policies under the same parser and training data.

The baseline that isolates the key ingredient is the identical surrogate and kernel run as global Bayesian optimization, with acquisition maximized over the whole cube. A second ablation is random search over policies. All search methods receive the same budget of 150 single-seed fine-tuning evaluations, the same initial design of 10 random policies, and the same final 5-seed test protocol. Hyperparameters of the downstream model are frozen across all arms, so only the policy varies.

Why not just…?

Why not just do greedy forward or backward selection over categories? This is the most serious competitor. If the optimum is close to additive, greedy selection will find it. One backward pass from full normalization costs 33 evaluations, well under our budget. We run it as a baseline under the same budget, allowing repeated greedy passes. The interaction analysis then decides the question: if greedy matches trust-region search and the fitted interaction terms are negligible, we will report that interactions are weak. That would itself be a useful negative finding, and we would recommend greedy selection.

Why not just fine-tune on raw text and let the model learn what matters? This is the 0.511 versus 0.589 gap. With 5,000 training posts, BERT-large does not learn to discount spelling noise on its own, which is why manual normalization helped so much. A byte-level model might narrow the gap, so we add raw-text ByT5 fine-tuning as an extra reference. If it matches the optimized policy, selective normalization is unnecessary for that model family.

Why not just prompt an LLM to normalize while preserving emotional tone? A Chat About Boring Problems finds that GPT-4 normalization still makes meaning-altering errors and drifts in style. AKCIT-FN at CheckThat! 2025 finds that fine-tuned small models beat LLM prompting when supervised data exist, as they do here. We nonetheless include an "affect-preserving" LLM normalization prompt as a baseline. If it reaches the searched policy's QWK, the search adds little beyond interpretability.

Proposed Experiments

The decisive experiment runs on WRIME in the oracle setting. We compare trust-region search, global search, random search, greedy selection, the all-but-pronunciation heuristic, and the two endpoints (no normalization and full normalization) on 5-seed test QWK and on best-so-far validation QWK as a function of evaluation count. The analysis of the Black-Box Optimization Challenge 2020 found that Bayesian optimization beats random search on hyperparameter tuning. Our analogous prediction is that trust-region search reaches the full-normalization score in far fewer than 150 evaluations and then exceeds it.

The second experiment repeats the comparison with predicted BART edits, testing whether the oracle policy carries over to realistic normalization.

The third experiment runs the search on MultiLexNorm for at least four languages, including one code-switched pair, with parsing as the objective. We then cross-apply policies: the parsing-optimal policy is scored on sentiment and vice versa. The task-dependence claim is supported if each cross-applied policy loses to the policy optimized for its own task by more than seed noise.

The project is falsified on WRIME if trust-region search fails to beat both the all-but-pronunciation heuristic and greedy selection on test.