Abstract
Large Language Models (LLMs) often struggle with low-resource languages due to data scarcity. This paper introduces TALL (Trainable Architecture for Enhancing LLM Performance in Low-Resource Languages), a novel approach that integrates frozen translation models with a frozen high-resource LLM. By training lightweight alignment adapters and custom transformer layers, TALL translates low-resource inputs into high-resource latent representations. Experimental results on Hebrew datasets demonstrate that TALL significantly outperforms standard fine-tuning and naive text-translation methods, achieving up to an 8.8x improvement in accuracy for models with minimal prior exposure to the target language.
Introduction
While modern LLMs exhibit remarkable capabilities in high-resource languages like English, their performance drops significantly for low-resource languages. Common remedies such as cross-lingual transfer, standard fine-tuning, or "naive translation" (text-to-text translation before and after LLM processing) often fall short. Fine-tuning is frequently hampered by data scarcity and linguistic complexity, while naive translation can lose structural nuance and semantic richness.
The TALL architecture addresses these challenges by structurally integrating translation models directly into the LLM's architecture. This allows the system to process low-resource languages natively in the embedding space without requiring the retraining of the massive underlying LLM.
Methodology
The TALL architecture employs a parameter-efficient "sandwich" design that routes data through a seven-stage pipeline. Instead of translating raw text, TALL operates on embeddings to preserve linguistic features.
The pipeline consists of:
- Frozen Translation Encoder: A pre-trained Marian MT model encodes the low-resource input (e.g., Hebrew) into an intermediate semantic state.
- Trainable Adapters & Custom Transformers: Multi-Layer Perceptron (MLP) adapters reshape the data to match the LLM's dimensionality. A custom cross-attention transformer then enriches these representations.
- Frozen LLM: A pre-trained high-resource LLM (such as Bloomz or Qwen) processes the aligned representation.
- Trainable Output Alignment: The LLM's output passes through another set of trainable adapters and a custom transformer.
- Frozen Decoder: A frozen Marian MT decoder generates the final low-resource output text.
This approach is highly parameter-efficient; only approximately 13–14% of the total system parameters are updated during training, while the massive pre-trained components remain frozen.
Experimental Setup
The architecture was trained on 256,000 Hebrew sentences from news articles. Evaluation was conducted on two distinct datasets to test generalization:
- WIKI-SLVM: 50,000 sentences from the Hebrew Wikipedia Corpus.
- HE-LYRICS: 50,000 sentences from a Hebrew Stage and Lyrics dataset.
The primary task was predicting the exact missing final word of a Hebrew sentence, a rigorous test of linguistic understanding.
Results and Discussion
The experimental results highlight the effectiveness of the TALL architecture across different base models and datasets.
Key findings include:
- Significant Performance Gains: For the Bloomz-560m model, which has minimal prior Hebrew exposure, TALL improved accuracy from 0.63% (direct) to 5.59%—an 8.8x increase. For QWEN2.5-0.5b, which has moderate prior exposure, TALL still provided a substantial boost to 5.15% accuracy.
- Failure of Standard Fine-tuning: Interestingly, standard fine-tuning on the target language actually degraded performance compared to the zero-shot baseline (e.g., dropping from 0.63% to 0.16% for Bloomz). This underscores the need for architectural interventions rather than simple data augmentation.
- Latent vs. Naive Translation: TALL consistently outperformed naive text-based translation, suggesting that alignment in the continuous embedding space better preserves context and linguistic nuance.
- Cross-Domain Robustness: Despite being trained on news data, TALL maintained its performance on Wikipedia and lyrics, demonstrating strong generalization capabilities.
Limitations and Future Work
Despite its success, TALL faces several challenges. The multi-stage pipeline increases inference latency and complicates standard optimizations like KV caching. Additionally, the method depends on the availability of high-quality bilingual translation models. Current training is also limited to a single-token learning signal (predicting the final token), rather than full auto-regressive training.
Future research will explore non-linear alignment techniques to mitigate information bottlenecks, extend the architecture to more languages, and optimize the pipeline for faster inference.
Summary updated