TL;DR
SENSIA introduces a linguistically motivated approach to multilinguality by explicitly aligning latent "sense" representations across languages using a Backpack language model architecture. This highly data-efficient method outperforms traditional dense-representation alignment and matches from-scratch monolingual models while using 2–4× less target-language data.
What Problem Does This Paper Address?
Modern multilingual language models (like XLM-R or BLOOMZ) typically achieve cross-lingual capabilities implicitly through massive parameter sharing and scale. However, we still lack a clear understanding of how this cross-lingual competence arises, and the resulting alignment can be uneven across layers or brittle across diverse languages.
Most existing approaches are engineering-driven—relying on massive compute and data without explicitly defining what units of meaning should be aligned. The authors address this gap by proposing a linguistically grounded alternative: a model should be multilingual because it represents similar meanings with comparable internal "senses" across languages, rather than just sharing a parameter space.
Methods
The authors introduce SENSIA (SENse-based Symmetric Interlingual Alignment). It builds on the Backpack language model architecture, which replaces standard token embeddings with a structured sense layer. Instead of a single dense vector, a Backpack represents each word as a context-dependent soft mixture of $K$ latent sense vectors.
SENSIA adapts a pretrained English Backpack model to a target language using parallel data via three complementary objectives:
- Sense Loss: A symmetric InfoNCE contrastive loss that aligns the sentence-level sense mixtures across the source and target languages.
- Context Loss: An InfoNCE loss that aligns the contextualized transformer representations (the final hidden states of the last non-pad tokens).
- Language Modeling (LM) Loss: A target-side cross-entropy loss to preserve fluency.
Training is executed via a smooth three-phase curriculum (Alignment $\rightarrow$ Joint $\rightarrow$ Polish) controlled by a cosine-based weight schedule. This transitions the model from establishing cross-lingual correspondences to refining target-language fluency while freezing the sense space to prevent representational drift.
Datasets
- Training & Adaptation: Sentence-parallel bitext automatically filtered and compiled from multiple OPUS corpora (capped at 2 million aligned pairs per language).
- Evaluation:
- FLORES-200: Used to track intrinsic training dynamics like bidirectional retrieval recall@1, sense-weight entropy, and perplexity.
- Downstream Tasks: Evaluated on XCOPA (causal reasoning), XStoryCloze (narrative cloze), and Belebele (reading comprehension) across four typologically diverse languages: Estonian, Indonesian, Swahili, and Turkish.
Key Insights and Ideas
- Meaning Preservation over Memorization: Through Sense Topology and Procrustes analyses, the authors demonstrate that SENSIA preserves both the local geometric structure and global manifold of the original English sense space. The model reorganizes meaning rather than simply memorizing surface-level lexical correspondences.
- The Power of Soft Mixtures: Ablation studies show that forcing the model to use only the top-1 most probable sense drastically degrades performance (increasing token-level cross-entropy by ~6 points). Softly combining multiple latent senses per token is critical for fluent prediction.
- Graceful Low-Data Degradation: SENSIA is remarkably robust in low-resource settings. When the training bitext is downsampled by 20× (to just 100k sentence pairs), macro accuracy across downstream tasks drops by only about 3 percentage points.
Key Results
- High Data Efficiency: SENSIA achieves competitive accuracy against monolingual models trained from scratch (the Goldfish baseline) while requiring 2.0× to 3.9× less target-language data.
- Outperforming Dense Baselines: SENSIA consistently outperforms standard dense-representation Multilingual Contrastive Learning (MCL) by an average of +2.3 percentage points (pp) and standard fine-tuning by +3.56 pp.
- Stable Alignment: Intrinsic metrics show that bidirectional retrieval recall@1 on FLORES-200 rapidly converges to nearly 0.97 for most languages, proving that sense adaptation creates a stable, non-collapsing bilingual mapping.
Limitations and Future Work
- Hostile Tokenizers: SENSIA currently relies on a shared English-centric BPE tokenizer. A stress test on Chinese revealed that when a target language is severely over-segmented into poorly aligned subwords, sense adaptation cannot fully compensate, trailing the from-scratch baseline by 3–4 points.
- Short-Form Supervision: The parallel sentences from OPUS are typically short and literal, which limits the model's exposure to broader world knowledge and discourse-level reasoning.
- Future Directions: The authors plan to scale sense adaptation to larger LLM backbones (7B+ parameters), explore lightweight target-specific tokenizer adaptations for non-Latin scripts, and experiment with document-level parallel corpora or parallel instruction data.
Summary updated