Harnessing Test-time Adaptation for NLU tasks Involving Dialects of English

Duke Nguyen, Aditya Joshi, Flora D. SalimarXiv 2025 · arXiv · OpenReview

TL;DR

This paper explores Test-Time Domain Adaptation (TTDA) using the SHOT technique to improve Natural Language Understanding (NLU) models on English dialects. They introduce the concept of "dialectal gap" and find it positively correlates with TTDA's effectiveness, suggesting TTDA is a valuable tool when labeled dialectal data is scarce.

What Problem Does This Paper Address?

Natural language exhibits significant variation across different dialects of English (e.g., Indian English, Singaporean English). NLU models, often trained on Standard American English (SAE), struggle to generalize to these diverse dialects. This "dialectal gap" is exacerbated by the scarcity of labeled data for many English dialects, making it difficult to fine-tune models for each specific variety. Existing approaches often rely on creating new datasets or using complex adapter mechanisms, which can be resource-intensive. This paper aims to address this challenge by investigating whether TTDA, a method that adapts models during inference without labeled target data, can effectively bridge the dialectal gap.

Methods

The core of this paper's methodology is the application of SHOT (Source Hypothesis Transfer), a well-established TTDA technique, to dialectal NLU tasks. SHOT works by leveraging unlabeled target domain data during inference to adapt a model trained on a source domain. Specifically, it involves:

  1. Source Model Training: A base model (roberta-base in this case) is trained on a source dialect (SAE or a specific dialect) with label smoothing. This model consists of an embedding network (F), a bottleneck network (BN), and a classifier (C).
  2. Information Maximization: The model's parameters are adjusted using unlabeled target dialect data to maximize information flow between the input and the classifier's output. This is achieved through a combination of losses, including a pseudo-labeling strategy where the model generates its own labels for the target data.
  3. Weighted k-means Clustering: An optional step to further refine the pseudo-labels by clustering embeddings from the bottleneck network.

The authors also introduce the concept of "dialectal gap," a measure of the performance difference between a model trained on a specific dialect and evaluated on that same dialect versus its performance on a different dialect. They hypothesize and test a positive correlation between this gap and the effectiveness of TTDA.

Datasets

The paper utilizes existing English dialectal datasets generated using Multi-VALUE (Ziems et al., 2023). Multi-VALUE is a system designed to create dialectal versions of standard NLP tasks by applying documented syntactic variations of dialects. The authors generated dialectal versions of the GLUE benchmark tasks (CoLA, RTE, SST-2) for four English dialects:

  • SAE (Standard American English)
  • IndE (Indian English)
  • NgE (Nigerian English)
  • SingE (Singaporean English)

These generated datasets were used for both training and evaluation.

Key Insights and Ideas

  • TTDA as a Dialect Bridge: The study demonstrates that TTDA, specifically SHOT, is a viable and effective technique for improving NLU model performance on unseen English dialects, especially when labeled data is scarce.
  • The "Dialectal Gap" Matters: The introduction and empirical validation of the "dialectal gap" concept are significant. The finding that a larger dialectal gap leads to greater benefits from TTDA provides a theoretical framework for understanding when TTDA is most useful.
  • SAE as a Strong Source: Surprisingly, fine-tuning on SAE and then applying TTDA to a target dialect often yields better results than fine-tuning directly on the target dialect itself. This suggests that SAE-trained models, when augmented with TTDA, can be more robust than models that only see limited dialectal data during fine-tuning.
  • Cross-Dialectal Fine-tuning Nuances: The paper notes that in some cases, fine-tuning on one dialect and then evaluating on another (cross-dialectal fine-tuning) can outperform fine-tuning on the target dialect. This highlights the complex interactions between dialects and model adaptation.

Key Results

  • SHOT consistently improves performance: When models are fine-tuned on SAE, SHOT consistently improves performance when adapting to target dialects, often outperforming models that were fine-tuned directly on the target dialect.
  • Positive correlation between dialectal gap and TTDAgain: The study found a statistically significant positive correlation (p-value < 0.05) between the dialectal gap and the performance improvement gained from TTDA (TTDAgain). This correlation was stronger when the source dialect was SAE.
  • SAE fine-tuning advantage: In many instances, fine-tuning on SAE and then applying TTDA resulted in higher performance compared to fine-tuning directly on the specific dialect. For example, on CoLA, RTE, and SST-2, the average TTDAgain when the source was SAE was 0.98, outperforming some direct fine-tuning scenarios.

Limitations and Future Work

  • Limited Dialect Coverage: The study acknowledges that Multi-VALUE, while useful, generates dialectal variations primarily through syntactic transformations and may not fully capture the nuances of natural dialectal variation.
  • Task Limitations: Due to computational constraints and the nature of the SHOT algorithm, the study could not evaluate on all Multi-VALUE GLUE tasks, specifically excluding STS-B (a regression task) and QQP.
  • Future Directions: The authors suggest exploring more sophisticated TTDA techniques and expanding the evaluation to a wider range of dialects and NLP tasks. Further research into how to better capture natural dialectal variation beyond syntactic changes is also implied.

Summary updated

Abstract

Test-time domain adaptation (TTDA) is an excellent method which helps generalize models across domains, tasks, and distributions without the use of labeled datasets. Thus, TTDA is very useful in natural language processing (NLP) in the dialectal setting, since oftentimes, models are trained on Standard American English (SAE), evaluated on Indian English (IndE), Singaporean English (SingE), or Nigerian English (NgE), of which distribution differs significantly from the former. This is especially useful since dialectal datasets are scarce. In this paper, we explore one of the most famous TTDA techniques, SHOT, in dialectal NLP. We finetune and evaluate SHOT on different combinations of dialectal GLUE. Our findings show that SHOT is a viable technique when labeled datasets are unavailable. We also theoretically propose the concept of dialectal gap and show that it has a positive correlation with the effectiveness of SHOT. We also find that in many cases, finetuning on SAE yields higher performance than finetuning on dialectal data.
BibTeX
@misc{nguyen2025harnessing,
  author = {Duke Nguyen and Aditya Joshi and Flora D. Salim},
  title = {{Harnessing Test-time Adaptation for NLU tasks Involving Dialects of English}},
  year = {2025},
  eprint = {2503.12858},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.12858},
}
Figure