Choosing a Prompt Scaffold per Input for Low-Resource Emotion Detection

In languages such as Amharic, Afan Oromo, Tigrinya and Somali, a small multilingual encoder adapted to that region and fine-tuned on a few thousand labelled examples still beats large chat models at detecting emotions in text. Prompting tricks do help the large models on some inputs. Translating the text to English first, for example, cuts errors on Marathi. On other inputs the same trick hurts: translation can erase sarcasm and cultural cues, so the model reads sarcasm as literal anger. No single prompting recipe wins everywhere, and each one costs extra tokens.

We propose to learn, for each input, which of a small menu of options to use. The options are: trust the cheap encoder; ask a large model directly in the native language; ask it to translate first; ask it to separate literal content from intended meaning; or ask it to check its own labels. The choice is trained against three separate goals rather than one blended score: accuracy on common emotions, recall on rare emotions such as disgust, fear and surprise, and token cost. Because the goals stay separate, practitioners get a menu of operating points, each a trade-off between rare-emotion accuracy and inference spend, rather than a single system.

Core Claim

Our claim is that the per-input choice among prompt scaffolds carries real, recoverable gains over (a) the best fixed scaffold and (b) a simple "escalate when the encoder is unsure" cascade, specifically on rare emotion classes. We also claim that the best choice shifts with the size of the large model. The first claim is falsifiable before any learning happens. If an oracle that picks the best option per input does not beat the best fixed option by a meaningful margin, the idea is dead and we report that.

The second claim follows from evidence that structured prompting needs capacity. TADIS has models label their demonstration examples as right or wrong before answering. That gives almost nothing at 770M parameters but large gains at 3B–7B. RAGLAB finds that self-critiquing retrieval only pays off at 70B rather than 8B. We therefore predict that a router trained over an 8B model will rarely buy decomposition or verification, while one trained over a 70B model will buy them often.

Positioning Against Prior Work

The multi-objective prompt optimization papers supply our framing but solve a different problem. Pareto Prompt Optimization trains a small prompt generator with preference losses that map out an accuracy–fluency trade-off on English classification. Efficient Multi-objective Prompt Optimization via Pure-exploration Bandits picks Pareto-optimal prompts from a fixed pool for English summarization, trading ROUGE-L against brevity. Both choose one prompt for a whole dataset. We choose per input, include a non-prompting option (the encoder), and work in languages where the large model itself is unreliable.

Two methods supply mechanisms we reuse. Tailoring Self-Rationalizers with Multi-Reward Distillation (MARIO) conditions a 0.7B model on control tokens for each reward so that one model can be steered along several objectives. We borrow the conditioning idea for the router, but we need no 175B teacher, which matters because large teachers are exactly what fails in these languages. DORB normalizes rewards by quantiles so that one reward cannot swamp the others. We reuse that normalization because token cost and per-label correctness live on very different scales.

The low-resource work motivates the gap and also constrains the design. Evaluating the Capabilities of Large Language Models for Multi-label Emotion Understanding introduces EthioEmo and shows that the fine-tuned encoder beats zero-shot and few-shot Llama-3, Gemma and GPT-4o-mini. It also shows translate-then-classify is "not always beneficial." The encoder therefore has to be the default option, not something the large model replaces. Chain-of-Translation Prompting (CoTR) shows the opposite pattern on Marathi, where translating inside the prompt helps even an 8B model. These two papers seem to contradict each other. Our view is that they describe different inputs, and that the contradiction is the routing signal.

UncleLM already mixes a fine-tuned XLM-RoBERTa with RAG-guided GPT-4o-mini prompting, plus an ad hoc "Are you sure?" re-prompt. It does not learn when to call the LLM or which prompt to use, and it still fails on "surprise." Finally, When Splitting Makes Stronger gives the basis for a decomposition scaffold: split a task into independent sub-questions and merge the answers. We apply it in a narrow form, asking for literal content and intended stance separately.

Research Program

We first build the option menu. One option is an AfroXLMR-76L encoder fine-tuned with the compound focal-plus-weighted-cross-entropy loss of LATE-GIL-NLP, with per-label thresholds tuned on development data as in UncleLM. The other four are prompt scaffolds run on a large model at two scales, Llama 3.1 8B and 70B, plus GPT-4o-mini:

  • native-language direct labelling,
  • CoTR-style translate-first labelling,
  • a two-part decomposition that asks for literal content and for intended stance before labelling,
  • a verification pass in which the model re-examines each emotion it assigned.

Every option is run once on every training and development instance and its outputs are logged. Because all outcomes are observed, router training is cost-sensitive supervised learning, not online exploration. That is cheaper and avoids bandit instability. The router is a small classifier whose inputs are three things: the encoder's hidden state and label probabilities, a language identifier, and a preference vector. Its output is a distribution over the five options.

For an input $x$ and option $a$, we define three normalized rewards:

  • $r_{\text{freq}}(x,a)$: the count of correct label decisions on frequent emotions,
  • $r_{\text{rare}}(x,a)$: the same count on disgust, fear and surprise,
  • $r_{\text{cost}}(x,a)$: the negative token count.

Each reward is quantile-normalized per language following DORB. The router is trained to maximize the expected value of

$$R_w(x,a) = w_1 r_{\text{freq}}(x,a) + w_2 r_{\text{rare}}(x,a) + w_3 r_{\text{cost}}(x,a),$$

where $w=(w_1,w_2,w_3)$ is the preference vector. During training $w$ is sampled at random, and the router is conditioned on it, as in MARIO's control tokens. At test time, sweeping $w$ traces a front of operating points from a single trained router.

We keep three objectives rather than one per emotion on purpose. Pareto Prompt Optimization reports that dominance-based methods lose their signal as the number of objectives grows. Grouping rare labels into one objective keeps the front meaningful. Per-input correctness counts are a surrogate for set-level macro-F1, so the front is always evaluated with true macro-F1 and per-label F1 on held-out data.

Baselines

The do-nothing reference is the fine-tuned AfroXLMR-76L encoder with tuned thresholds, used for every input. It is also the strongest existing method on EthioEmo. Evaluating the Capabilities of Large Language Models for Multi-label Emotion Understanding reports weighted F1 of 72.6% on Afan Oromo and 69.9% on Amharic for AfroXLMR-76L. On the SemEval-2025 Track A data, the adaptive-pretraining system of NUST Titans reports Macro-F1 of 0.71 on Amharic.

We will retrain this encoder ourselves on the same splits, with the same loss and threshold search and the same tuning budget, so that the router's default option is no weaker than the published bar. Beating it means three things together:

  • higher held-out macro-F1 than the encoder at some budget,
  • a statistically significant gain in rare-class F1 across bootstrap resamples,
  • the extra token cost reported at that point.

Showing gains only at unlimited budget does not count. Because the router can always choose the encoder, it should never fall below this baseline. If it does, the router is broken.

The best fixed scaffold is the single prompt option with the highest development macro-F1 per language and model size. It is evaluated at its own token cost, and the router is compared at matched cost. On Marathi, the published CoTR result for Llama 3.1 8B (average classification error rate 23.84% against 29.13% for standard prompting) lets us check that our CoTR replication is faithful before it enters the menu. For GPT-4o-mini zero-shot on EthioEmo there is no reported numeric result on Lacuna; the paper says only that it "trailed behind the specialized encoder models." We will measure it.

The confidence cascade is the main ablation. It escalates to the best fixed scaffold whenever the encoder's label entropy exceeds a threshold, and the threshold is swept to give a cost–accuracy curve. It isolates our key ingredient, choosing which scaffold per input, from the simpler ingredient of choosing whether to escalate.

A scalarized grid baseline trains separate unconditioned routers for a grid of fixed weight vectors, with the same total training compute. It tests whether preference conditioning adds anything over brute force.

All arms use the same logged option outputs, the same data splits, and the same number of hyperparameter trials.

Why not just…?

Why not just fine-tune the encoder better and skip large models entirely? This is the most serious threat. The EthioEmo results and the NUST Titans adaptive-pretraining ablation (about a 10% Macro-F1 drop without it) show that encoder work pays off. Our design absorbs this alternative instead of competing with it: the strongest encoder is the default option, so any router gain is gain on top of it. If the oracle-headroom check shows the large-model options almost never fix encoder errors on rare emotions, we will report that the encoder is sufficient and stop.

Why not just escalate uncertain inputs to one good prompt? This may well capture most of the benefit, and the confidence cascade baseline exists to settle it. The case for per-input choice rests on the CoTR versus EthioEmo tension: translation helps on some inputs and destroys sarcasm on others. The experiment that decides it compares the router and the cascade at matched token budgets. If the cascade comes within the bootstrap confidence interval, we will conclude that scaffold choice does not matter and recommend the cascade.

Why not just use a weighted sum with fixed weights? With only three objectives, a grid of fixed weights may recover most of the front, and we say so. Pareto Prompt Optimization reports that summation and product scalarization collapse toward single points, but in a prompt-generation setting with a much larger search space. We therefore treat the scalarized grid as a genuine competitor. We keep preference conditioning only if it matches the grid's held-out hypervolume (the area of objective space dominated by the front) at lower training cost.

Why not just call a bigger model? The EthioEmo study found few-shot gains "inconsistent or plateaued quickly." The SemEval-2025 Task 11 overview, SemEval-2025 Task 11, reports zero-shot cross-lingual F-scores "often below 0.50 in many languages." We nonetheless include 70B and GPT-4o-mini options, so scale is tested directly rather than assumed away.

Proposed Experiments

The data are EthioEmo (Amharic, Afan Oromo, Somali, Tigrinya) and the low-resource Track A languages of SemEval-2025 Task 11, including Tigrinya, Igbo, Oromo and Amharic. The Marathi MahaSent and MahaHate sets serve as a setting where translation is known to help. We report:

  • weighted F1, for comparability with published numbers,
  • macro-F1,
  • per-label F1 for disgust, fear and surprise,
  • tokens per instance,
  • hypervolume of the held-out front.

The first decisive check is oracle headroom. Before training any router, we compute the per-input best option on development data and compare it with the encoder and with the best fixed scaffold. We also measure how often the best option differs across inputs. If the oracle's gain on rare classes is small, or the same option is nearly always best, the proposal is withdrawn.

The second check is matched-budget comparison. We compare the router's front against the cascade's curve and the scalarized grid at equal token spend. Significance comes from bootstrap resampling over test instances.

The third check is the scale shift. We train routers over 8B and over 70B options separately and test whether the 70B router chooses decomposition or verification significantly more often, as TADIS and RAGLAB predict. The Open Ko-LLM Leaderboard study finds that skills only begin to reinforce one another at 7–14B. That leads us to expect the 8B router to rely more on the encoder.

The fourth check is a targeted error analysis. We annotate a held-out sample of sarcastic and idiomatic items and test whether the router avoids translate-first on exactly the items where translation flips the label.

Finally, to probe transfer, we train the router on three EthioEmo languages and apply it to the fourth, in the spirit of Track C. This tests whether the routing signal comes from the encoder's uncertainty and the input's features rather than from memorized language identity.