Compression Effects on Low-Resource Multilingual Translation

Abbas SaniarXiv 2026 · arXiv

TL;DR

This pilot study investigates how post-training quantization (PTQ) affects machine translation for African languages (Hausa, Swahili, Yoruba) on consumer-grade hardware. The findings reveal a "latency paradox": while quantization saves up to 61% of VRAM, it actually increases inference time by up to 5.3x on NVIDIA T4 GPUs, even as translation quality remains remarkably stable.

What Problem Does This Paper Address?

As large language models (LLMs) grow, compression techniques like quantization—reducing the precision of model weights to save memory—have become essential for deployment. However, most research in this area focuses on high-resource languages (like English) and server-class hardware (like A100 GPUs).

This paper addresses the "deployment gap" for low-resource languages, which are often spoken in regions where consumer-grade hardware (like the NVIDIA T4) is the norm. The author asks two critical questions:

  1. Does aggressive quantization disproportionately harm translation quality for low-resource languages?
  2. Do the theoretical latency benefits of quantization actually materialize on the hardware typically available in these contexts?

Methods

The author evaluated three distinct model families:

  • Qwen2.5-1.5B-Instruct: A small, general-purpose LLM.
  • Aya-23-8B: A larger, multilingual instruction-tuned LLM.
  • NLLB-200-distilled-1.3B: A specialized, "No Language Left Behind" encoder-decoder model dedicated to translation.

These models were tested across three precision levels using the bitsandbytes library:

  • FP16: The native half-precision baseline.
  • INT8: 8-bit weight quantization.
  • NF4 INT4: 4-bit NormalFloat quantization with double quantization.

The hardware setup utilized NVIDIA Tesla T4 GPUs (16 GB VRAM), representing a common tier for accessible cloud compute and high-end consumer setups.

Table 1: Models evaluated in the study.

Datasets

The study used the FLORES-200 devtest set, a standard benchmark for multilingual translation. Evaluation focused on 1,012 parallel sentences per language across six directions: Hausa (hau), Swahili (swh), and Yoruba (yor) to/from English (eng).

Key Insights and Ideas

  • Specialization Trumps Scale: Despite being roughly one-sixth the size of the 8B Aya model, the specialized NLLB-1.3B model dominated the results. This suggests that for low-resource translation, targeted training data is far more important than raw parameter count or general instruction-tuning.
  • The Latency Paradox: In many research papers, quantization is synonymous with "faster inference." This study provides a reality check: on consumer GPUs, the overhead of dequantizing weights during every matrix multiplication actually makes the model slower than the FP16 baseline.
  • Quality Robustness: A common fear is that quantization "breaks" the fragile representations of low-resource languages. The data suggests otherwise; aggregate quality (measured by chrF) dropped by less than 1 point in most cases, indicating that modern quantization methods are surprisingly robust across resource levels.
  • The Resource Gradient: There is a clear performance ceiling dictated by training data availability. Swahili (the highest resource of the three) performed best, followed by Hausa, then Yoruba.

Key Results

  • Translation Quality: NLLB-1.3B outperformed the LLMs by massive margins: 32.7–43.3 chrF on Hausa and Swahili directions. For Yoruba, the gap was smaller but still decisive (12.9–24.7 chrF).
  • Quantization Stability: Across all models and directions, INT4 quantization preserved quality within approximately 1 chrF of the baseline, with the worst-case drop being only 2.19 chrF.
Table 3: chrF change relative to FP16. Negative values indicate quality loss.
  • Memory vs. Speed: INT4 provided 49–61% memory savings, allowing the 8B Aya model to fit on a single T4 GPU. However, this came at a steep cost: inference latency increased by 1.3× to 5.3× compared to FP16.
Table 4: Memory and latency across precisions. Note the significant slowdowns in the 'vs FP16' column.

Limitations and Future Work

The author acknowledges several limitations inherent in a pilot study:

  • Limited Compression Methods: The study only looked at bitsandbytes PTQ. Other methods like GPTQ, AWQ, or distillation might yield different latency profiles.
  • Hardware Specificity: The latency findings are specific to the T4 architecture; newer or more powerful accelerators might handle dequantization more efficiently.
  • Metric Sensitivity: Aggregate metrics like chrF can hide specific linguistic errors (e.g., mistranslating named entities or numerals). Future work should include fine-grained error analysis.
  • Yoruba Sweep: Due to compute constraints, the full quantization sweep was not completed for Yoruba, leaving a gap in the "tail" of the resource gradient.

The study concludes by calling for a research agenda focused on hardware-aware compression—developing methods that optimize for the specific bottlenecks of the consumer hardware used in low-resource regions.

Summary updated

Abstract

We present a pilot study characterising the effect of post-training quantisation on multilingual machine translation quality and inference cost for three African languages - Hausa, Swahili, and Yoruba - across three model families: a small general-purpose LLM (Qwen2.5-1.5B), a larger multilingual instruction-tuned LLM (Aya-23-8B), and a dedicated multilingual MT model (NLLB-200-distilled-1.3B). Evaluating on the FLORES-200 devtest set (1,012 sentences per direction) under FP16, bitsandbytes INT8, and NF4 INT4 precisions on consumer-grade NVIDIA T4 GPUs, we find: (i) the specialised MT model outperforms both LLMs by 32.7-43.3 chrF on the Hausa and Swahili directions, and by 12.9-24.7 chrF on Yoruba, despite being roughly a sixth the size of the 8B instruction-tuned LLM; (ii) post-training quantisation preserves aggregate translation quality within approximately 1 chrF across all model families and language directions, with a single worst case of 2.19 chrF; (iii) on consumer GPUs, INT8 and INT4 quantisation provide substantial memory savings (49-61% at INT4) but do not reduce inference latency relative to FP16 - in fact, latency increases by 1.3x to 5.3x; (iv) a clear performance gradient tracking training-resource availability is visible across languages (Swahili > Hausa > Yoruba). These findings motivate a research agenda focused on hardware-aware compression and error-type analysis beyond aggregate metrics for low-resource deployment.
BibTeX
@misc{sani2026compression,
  author = {Abbas Sani},
  title = {{Compression Effects on Low-Resource Multilingual Translation}},
  year = {2026},
  eprint = {2611.24043},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24043},
}
Figure