Abstract
Cross-lingual transfer typically requires fine-tuning multilingual models on labeled task data. However, data privacy often prevents access to the original training sets of high-performing monolingual models. This paper introduces PLUGIN-X, a modular framework that activates cross-lingual capabilities in existing monolingual task models without requiring the original training data. By disassembling models and reassembling them with a cross-model connector trained via Heterogeneous Masked Language Modeling (HMLM), PLUGIN-X achieves significant performance gains on cross-lingual benchmarks like XNLI and XQuAD compared to data-free baselines.
Introduction
In Natural Language Processing (NLP), cross-lingual transfer aims to adapt models trained in high-resource languages (e.g., English) to other languages. The standard paradigm involves fine-tuning a pretrained multilingual model (like mBERT or XLM-R) on labeled task data. However, a significant real-world constraint is the Public-Model-In-house-Data (PMID) setting: high-performing models are often released, but the proprietary datasets used to train them remain private due to legal or policy restrictions.
Without access to the original training data, traditional transfer methods fail. PLUGIN-X addresses this by asking whether cross-lingual potential can be "activated" using only model weights and unlabeled text in various languages.
Methodology: The PLUGIN-X Framework
The authors propose a "plug-and-play" approach that treats neural network architectures as modular components. The process is divided into three distinct phases: disassembly, reassembly, and adaptation.
Disassembly and Reassembly
First, a monolingual "End-Task Model" and a pretrained "Multilingual Model" are disassembled into their constituent sub-modules. These are then reassembled into a new pipeline:
- Multilingual Encoder: The lower layers of the multilingual model, responsible for general linguistic features.
- Cross-Model Connector: A set of new Transformer layers that bridge the representation gap between the encoder and the task head.
- End-Task Module: The upper layers and task head of the original monolingual model, which preserve the specific task "knowledge."
Representation Adaptation via HMLM
To ensure these disparate modules can communicate, the authors introduce Heterogeneous Masked Language Modeling (HMLM). This technique uses unlabeled text (e.g., from the CCNet corpus) to train the Connector to map the multilingual encoder's outputs into the representation space expected by the task module.
Crucially, HMLM utilizes the monolingual model's vocabulary for the output predictions while using the multilingual model's vocabulary for the input tokens. This "heterogeneous" setup forces the Connector to learn the translation between the two internal "languages" of the models.
Experimental Setup and Results
The framework was evaluated on two major cross-lingual benchmarks: XNLI (Natural Language Inference across 15 languages) and XQuAD (Question Answering across 10 languages).
Performance Gains
PLUGIN-X demonstrated substantial improvements over previous data-free baselines:
- XNLI: While simple embedding mapping (EMBMAP) achieved only 33.4% accuracy (near random for a 3-class task), PLUGIN-X reached 62.9%.
- XQuAD: On question answering, PLUGIN-X achieved an F1 score of 47.0, whereas previous methods were largely non-functional, scoring between 1.1 and 7.0.
Representation Alignment
Analysis using $L_2$ distance and cosine similarity confirmed that the Connector effectively aligned the representations of different languages with the original English model's space. This supports the hypothesis that the primary barrier to zero-data transfer is representation mismatch rather than a lack of task-specific knowledge.
Discussion and Conclusion
PLUGIN-X leverages the "Universal Layer" hypothesis—the idea that middle layers of multilingual Transformers produce naturally aligned cross-lingual representations. By "plugging in" at these layers, the method achieves a form of model fusion that combines task expertise with linguistic breadth.
While PLUGIN-X currently lags behind "gold-standard" fine-tuning on private data, it establishes that cross-lingual transfer is viable even when training data is inaccessible. Future work may extend this modular approach to generative tasks and larger, more complex architectures like Large Language Models (LLMs).
Summary updated