Abstract
Low-Rank Adaptation (LoRA) has emerged as one of the most widely used parameter-efficient fine-tuning (PEFT) methods for adapting large language models (LLMs) to downstream tasks. While highly effective in single-task settings, it struggles to efficiently leverage inter-task knowledge in complex multi-task learning scenarios, often requiring substantial task-specific data to achieve optimal performance. To address this limitation, we introduce MeTA-LoRA, a two-stage optimization framework that significantly improves data efficiency in multi-task adaptation. In the first stage, task-specific LoRA adapters are learned using only a few samples from each involved dataset, enabling rapid adaptation without large-scale supervision. In the second stage, the shared LoRA adapter is updated by aggregating gradients from multiple tasks to promote knowledge transfer across tasks, further reducing data usage by leveraging common patterns. In both multi-task learning and multilingual learning scenarios, our method matches or surpasses the performance of traditional full-data LoRA fine-tuning approaches, while using significantly less task-specific data.
Introduction
While Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA have revolutionized how we adapt Large Language Models (LLMs) by updating only a small fraction of weights, they often remain "data hungry" in multi-task scenarios. Adapting a model to dozens of tasks simultaneously typically requires hundreds of thousands of labeled examples to achieve stable performance. For instance, state-of-the-art methods like HydraLoRA require over 320,000 examples for 43 tasks. This data efficiency gap makes multi-task adaptation prohibitively expensive for niche domains or low-resource languages. Furthermore, simple data mixing often leads to "task interference," where learning one skill degrades performance in another.
MeTA-LoRA addresses these challenges by framing fine-tuning as an iterative "learning to learn" process. This framework enables LLMs to achieve high performance across multiple tasks or languages using only a tiny fraction of the data required by traditional methods—often matching or exceeding the performance of models trained on full datasets while reducing training time by up to 85%.
Methodology
MeTA-LoRA utilizes a two-stage optimization framework consisting of an inner and outer loop to decouple rapid task adaptation from global knowledge aggregation.
Phase I: Task-Specific Adaptation (The "Inner Loop")
For a given batch of tasks, the model creates temporary, local LoRA adapters. These adapters are updated using a very small "support set" (ranging from 8 to 100 samples). This allows the model to "rehearse" adaptation to a specific task without permanently altering global weights.
Phase II: Meta-Knowledge Update (The "Outer Loop")
The model evaluates these temporary adapters on a separate "query set." It then aggregates the gradients from all these tasks to update a single, shared global LoRA adapter. To maintain computational feasibility for billion-parameter models, MeTA-LoRA employs a first-order approximation of Model-Agnostic Meta-Learning (MAML). This avoids the high computational cost of Hessian-vector products by effectively detaching the computational graph after local adaptation.
Experimental Setup
The framework was evaluated across several diverse scenarios:
- Flanv2 Subset: 43 tasks grouped into 10 clusters (NLU and NLG).
- Five-Task Dataset: A custom mix including GSM8K (math), QQP/CosmosQA (NLU), and SiQA/PiQA (commonsense reasoning).
- Bactrian-X: A multilingual dataset covering 52 languages.
- Benchmarks: Performance was measured using BBH (Big-Bench Hard), MMLU (Massive Multitask Language Understanding), and multilingual benchmarks like XCOPA and XWinograd.
Key Results and Insights
Performance vs. Data Efficiency
MeTA-LoRA demonstrates a "less is more" paradox where it often outperforms standard LoRA despite using significantly less data. On the LLaMA2-13B model (Flanv2), MeTA-LoRA achieved a BBH score of 46.32 using only 100 samples per task. In comparison, HydraLoRA scored 44.7 while requiring the full dataset. This suggests that the meta-learning structure acts as a regularizer, preventing the model from being confused by noise in massive multi-task datasets.
Multilingual and Computational Gains
In the 52-language Bactrian-X test, MeTA-LoRA (using 50 samples per language) outperformed models trained on the full dataset of 3.4 million pairs on three out of four major benchmarks. Beyond data savings, the framework is extremely time-efficient; the training time for a LLaMA2-7B model dropped from 17 hours with standard LoRA to just 3 hours with MeTA-LoRA due to faster convergence.
Conclusion
MeTA-LoRA provides a robust solution for data-efficient multi-task adaptation by decoupling task-specific learning from global knowledge updates. While the current implementation uses a first-order MAML approximation and focuses on LoRA, future work may explore higher-order gradients and other PEFT methods like Prefix-Tuning. Additionally, investigating sophisticated task-scheduling strategies could further enhance the efficiency of the meta-learning process.
Summary updated