Abstract
We propose a Bi-Directional Manifold Alignment (BDMA) that learns a non-linear mapping between two manifolds by explicitly training it to be bijective. We demonstrate BDMA by training a model for a pair of languages rather than individual, directed source and target combinations, reducing the number of models by 50%. We show that models trained with BDMA in the "forward" (source to target) direction can successfully map words in the "reverse" (target to source) direction, yielding equivalent (or better) performance to standard unidirectional translation models where the source and target language is flipped. We also show how BDMA reduces the overall size of the model.
Introduction
In the field of Natural Language Processing, word embeddings—vector representations that capture semantic meaning—are typically trained for a single language in isolation. To enable cross-lingual applications such as machine translation or cross-lingual information retrieval, these independent vector spaces (manifolds) must be aligned. The goal is to ensure that semantically equivalent words across different languages, such as "dog" in English and "chien" in French, are mapped to nearby locations in a shared embedding space.
Traditional Cross-Lingual Word Embedding (CLWE) alignment methods generally suffer from two major limitations. First, they often rely on simple linear transformations because deeper, non-linear models are prone to overfitting in this specific task. Second, these models are typically unidirectional; translating between a language pair in both directions requires two separate models. This approach is not only computationally inefficient but also fails to exploit the inherent symmetry of translation.
Bi-Directional Manifold Alignment (BDMA)
The researchers introduce Bi-Directional Manifold Alignment (BDMA), a framework designed to learn a single, reversible, and non-linear mapping between two language manifolds. By enforcing bijectivity (a one-to-one correspondence), BDMA ensures that the mapping is inherently stable and reversible.
Key Methodological Innovations
- Cycle Consistency Loss (CCL): Drawing inspiration from CycleGANs used in computer vision, the authors implement a loss function that penalizes the model if a word mapped from Language A to Language B and then back to Language A does not return to its original starting point.
- Weight Transposition: To ensure parameter efficiency, the "reverse" mapping ($B \to A$) utilizes the exact same weight matrices as the "forward" mapping ($A \to B$), but in their transposed form. This architectural constraint forces the model to learn a weight structure that is naturally invertible.
- Layerwise Orthogonal Loss: An additional penalty is applied to keep the weight matrices orthogonal. This helps preserve the geometric relationships and distances between words during the non-linear transformation.
- Hybrid Distance Metrics: The training process optimizes a combination of Cosine Similarity and Relaxed Cross-domain Similarity Local Scaling (RCSLS), which the authors found to be the most effective objective for stable alignment.
Experimental Setup
The BDMA framework was evaluated using the MUSE (Multilingual Unsupervised and Supervised Embeddings) dataset, which includes 110 bilingual dictionaries. The experiments covered a broad spectrum of language pairs, including high-resource languages (Spanish, French, German, Italian) and more challenging low-resource or distantly related pairs (Russian, Japanese, Hindi). Notably, the researchers specifically analyzed the impact of polysemous words—words with multiple meanings—finding that their inclusion during training actually improved the model's ability to generalize across the manifolds.
Results and Discussion
The empirical results demonstrate that non-linear models, specifically a 1-hidden layer Feedforward Network, can match or exceed the performance of state-of-the-art linear mappings when constrained by the BDMA framework.
Performance and Efficiency
- State-of-the-Art Accuracy: BDMA achieved results comparable to or better than established unidirectional baselines such as VECMAP and RCSLS across various language pairs. For instance, in high-resource pairs like English-Spanish, the model maintained high Precision@1 scores in both directions.
- Model Compression: Because a single BDMA network handles both translation directions ($A \to B$ and $B \to A$), it effectively halves the storage and memory requirements compared to traditional systems that require two separate models.
- Low-Resource Improvements: The benefits of non-linearity were particularly evident in distantly related languages. In the Japanese-to-English task, the 1-hidden layer BDMA network showed a significant improvement of approximately 2.82% over standard linear mappings.
- Symmetry and Stability: The bi-directional constraint acts as a powerful regularizer. The authors observed that performance in the forward and reverse directions remained remarkably consistent, typically within 0.5% of each other.
Conclusion and Future Work
BDMA represents a significant step forward in cross-lingual embedding alignment by proving that non-linear, reversible mappings are both feasible and highly efficient. While the model performed well across most benchmarks, some challenges remain, such as a slight performance sensitivity to the initial training direction and lower accuracy on structurally divergent languages like Hindi. Future research could investigate deeper invertible architectures, such as Normalizing Flows, to further refine the alignment of complex language manifolds without sacrificing the benefits of bi-directionality.
Summary updated