Few-shot Controllable Style Transfer for Low-Resource Settings: A Study in Indian Languages

Kalpesh Krishna, Deepak Nathani, Xavier Garcia, Bidisha Samanta, Partha Talukdar

Abstract

Style transfer is the task of rewriting a sentence into a target style while approximately preserving content. While most prior literature assumes access to a large style-labelled corpus, recent work has attempted "few-shot" style transfer using only 3-10 sentences at inference for style extraction. In this work, we study a relevant low-resource setting: style transfer for languages where no style-labelled corpora are available. We notice that existing few-shot methods perform this task poorly, often copying inputs verbatim. We push the state-of-the-art for few-shot style transfer with a new method modeling the stylistic difference between paraphrases. When compared to prior work, our model achieves 2-3x better performance in formality transfer and code-mixing addition across seven languages. Moreover, our method is better at controlling the style transfer magnitude using an input scalar knob. We report promising qualitative results for several attribute transfer tasks (sentiment transfer, simplification, gender neutralization, text anonymization) all without retraining the model. Finally, we find model evaluation to be difficult due to the lack of datasets and metrics for many languages. To facilitate future research, we crowdsource formality annotations for 4,000 sentence pairs in four Indic languages and use this data to design our automatic evaluations.

Introduction

Style transfer—the task of changing a sentence's "vibe" (such as formality or sentiment) while preserving its core meaning—typically requires massive datasets of labeled sentences. For most of the world’s languages, these datasets do not exist. While recent "few-shot" approaches attempted to use English models to extract style from just a handful of examples at runtime, these models often fail in low-resource settings. Instead of transferring style, they frequently copy the input sentence word-for-word. This occurs because standard models are trained to "denoise" text where words were randomly deleted; in low-resource scenarios, the model learns that the safest way to "fix" a sentence is to change nothing at all.

Methodology: The DIFFUR Approach

To address these limitations, we introduce DIFFUR (Difference Universal Rewriter). Our method moves away from simple denoising and instead models the stylistic difference between paraphrases. The architecture incorporates three primary innovations:

  1. Paraphrases as "Natural Noise": Rather than randomly dropping tokens, we use backtranslation to create natural-sounding paraphrases. This forces the model to learn how to re-insert stylistic markers into a grammatically correct but stylistically "neutral" sentence.
  2. Style Vector Subtraction: Standard models attempt to extract a single style vector. DIFFUR calculates the difference between the vector of an original sentence and its paraphrase. This isolates the stylistic "delta," effectively separating the "how" from the "what."
  3. Multitask Learning (DIFFUR-MLT): We fine-tune the model using a combination of the original denoising objective and our new difference-based objective to maintain the integrity of the model's internal style map.
Overview of the DIFFUR inference process, demonstrating how the style vector difference and the transfer amount knob (λ) influence the choice of formal vocabulary (e.g., Sanskrit vs. Persian words) in Hindi.

At inference time, a user provides a few examples of the target style. The model calculates the average difference and applies it to a new sentence. A scalar "knob" ($\lambda$) allows for fine-grained control over the intensity of the transformation.

Datasets and Benchmarking

Our research utilized large-scale multilingual and Indic-specific corpora, including mC4, Samanantar, and OPUS-100. A major contribution of this work is the creation of a new evaluation benchmark. We crowdsourced 4,000 sentence pairs across four Indic languages—Hindi, Bengali, Kannada, and Telugu—with human annotations for formality and semantic similarity. This dataset is intended to support future research in low-resource NLP.

Results and Discussion

DIFFUR demonstrates significant improvements over existing state-of-the-art (SOTA) methods like the Universal Rewriter (UR).

  • Performance Gains: In formality transfer for Hindi, DIFFUR-MLT achieved an aggregated score (r-AGG) of 78.1, a massive leap from the baseline of 30.4.
  • Reduced Copying: One of the most critical improvements was the reduction in verbatim copying, which dropped from approximately 45% in the baseline to just 4.4% in our model.
  • Controllability: The model responds predictably to the intensity knob ($\lambda$). As shown in our evaluations, increasing $\lambda$ successfully produces more formal lexical choices (such as Sanskrit-derived terms in Hindi) without sacrificing content similarity.
Quantitative comparison of DIFFUR against baseline models. DIFFUR-MLT significantly improves aggregate performance (r-AGG) while drastically reducing verbatim copying.

Beyond formality, we demonstrated successful "zero-shot" transfers for sentiment, text simplification, and anonymization without task-specific retraining. This suggests that by treating style as a delta, the model becomes significantly more versatile across different stylistic attributes.

Conclusion

While DIFFUR represents a significant step forward, evaluation remains a challenge due to the relative inaccuracy of automatic style classifiers in low-resource languages compared to English. Additionally, while the model excels at lexical swaps, complex syntactic transformations (such as those required for gender neutralization) remain an area for future improvement. Future work will focus on these structural changes and expanding controllable style transfer to a wider array of niche stylistic attributes.

Summary updated

Abstract

Style transfer is the task of rewriting a sentence into a target style while approximately preserving content. While most prior literature assumes access to a large style-labelled corpus, recent work (Riley et al. 2021) has attempted "few-shot" style transfer using only 3-10 sentences at inference for style extraction. In this work we study a relevant low-resource setting: style transfer for languages where no style-labelled corpora are available. We notice that existing few-shot methods perform this task poorly, often copying inputs verbatim. We push the state-of-the-art for few-shot style transfer with a new method modeling the stylistic difference between paraphrases. When compared to prior work, our model achieves 2-3x better performance in formality transfer and code-mixing addition across seven languages. Moreover, our method is better at controlling the style transfer magnitude using an input scalar knob. We report promising qualitative results for several attribute transfer tasks (sentiment transfer, simplification, gender neutralization, text anonymization) all without retraining the model. Finally, we find model evaluation to be difficult due to the lack of datasets and metrics for many languages. To facilitate future research we crowdsource formality annotations for 4000 sentence pairs in four Indic languages, and use this data to design our automatic evaluations.
BibTeX
@inproceedings{krishna2022few,
  author = {Kalpesh Krishna and Deepak Nathani and Xavier Garcia and Bidisha Samanta and Partha Talukdar},
  title = {{Few-shot Controllable Style Transfer for Low-Resource Settings: A Study in Indian Languages}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=enfA6ONVYHT},
}
Figure