MeTA-LoRA: Data-Efficient Multi-Task Fine-Tuning for Large Language Models

Bo Cheng, Xu Wang, Jinda Liu, Yuan Wu, Yi ChangSubmission to ICLR 2026 · Rejected

TL;DR

While Low-Rank Adaptation (LoRA) dramatically slashes the parameter count needed to tune large language models, existing multi-task adaptation methods still require hundreds of thousands of training examples. META-LORA bridges this gap by applying a first-order meta-learning framework (MAML) to LoRA tuning, matching or surpassing full-data baselines using as few as 50 to 100 examples per task while cutting training time by over 80%.

What Problem Does This Paper Address?

Parameter-efficient fine-tuning (PEFT) methods like LoRA allow developers to adapt massive LLMs without fine-tuning billions of parameters. However, parameter efficiency does not equal data efficiency. When scaling to multi-task regimes, existing architectures (such as HydraLoRA or LoRAMoE) often demand hundreds of thousands of labeled samples—for instance, over 320,000 examples across 43 tasks—to generalize well.

Standard data-efficiency heuristics like coreset selection or data pruning cherry-pick high-utility samples for single tasks, but they often strip away the shared, cross-task structures essential for multi-task generalization. The authors tackle this fundamental trade-off: How can an LLM learn a shared multi-task LoRA adapter using only a tiny fraction of supervision per task?

Methods

Inspired by Model-Agnostic Meta-Learning (MAML), META-LORA formulates multi-task parameter adaptation as an episodic, two-stage optimization loop:

Shared LoRA (θ)
   │
   ├── [Phase 1: Task Adaptation] ──> Clone θ to local θ_i; take k steps on Support Set S_i
   │
   └── [Phase 2: Meta Update]    ──> Compute loss on Query Set Q_i; aggregate gradients
                                      Update shared θ: θ ← θ - β · (1/n) Σ ∇ L_{Q_i}(θ_i)
  1. Phase I: Task-Specific Adaptation (Inner Loop): In each iteration, $n$ tasks are sampled. For each task $T_i$, a temporary copy of the shared LoRA parameters $\theta$ is updated for $k$ gradient descent steps on a small support set $S_i$ (e.g., $n_s = 8$ samples, $k = 3$).
  2. Phase II: Meta-Knowledge Update (Outer Loop): The adapted local adapters $\theta_i$ are evaluated on a held-out query set $Q_i$ ($n_q = 8$). Gradients are detached from the inner loop to adopt a first-order MAML approximation, avoiding expensive second-order Hessian-vector calculations. The shared adapter $\theta$ is then updated via the AdamW optimizer using the mean gradient across all sampled tasks.
  3. Inference Simplicity: Unlike Mixture-of-Experts (MoE) methods that require complex task routers during inference, META-LORA outputs a single shared adapter that merges directly into the frozen base model weights ($W = W_0 + BA$), adding zero inference latency or architectural complexity.

Datasets

The authors evaluated META-LORA across both multi-task and cross-lingual settings using LLaMA2 (7B and 13B):

  • Flanv2 Benchmark: 43 tasks across 10 clusters (NLU, generation, translation, commonsense, QA). Evaluated on Big-Bench Hard (BBH) (3-shot).
  • Five-Task Diverse Benchmark: Curated to test diverse reasoning domains using GSM8K (math), QQP and CosmosQA (NLU), and SiQA and PiQA (commonsense). Evaluated on MMLU (5-shot) and BBH (3-shot).
  • Challenging Five-Task Variant: PiQA was substituted with WinoGrande to test more difficult reasoning.
  • Multilingual Benchmark: 52 languages from Bactrian-X (3.4M translated instructions). Zero-shot evaluation across four multilingual suites: XCOPA, XStoryCloze, XWinograd, and EXAMS.

Key Insights and Ideas

  • The Multi-Task LoRA Paradox: Standard LoRA often performs worse when fine-tuned on the full multi-task dataset than when fine-tuned on just 50–100 samples per task (e.g., LLaMA2-7B achieves 28.70% average on the full 5-task dataset vs. 36.75% with 50 examples/task). Because a single LoRA module has low intrinsic capacity, massive multi-task data causes task gradients to conflict and cancel each other out.
  • Optimization Beats Data Heuristics: When compared against sophisticated sample-selection strategies like BM25 and DSIR, random sampling with META-LORA still outperformed LoRA tuned on curated data (39.00% vs. ~36.3% on LLaMA2-7B). The gains stem from the meta-optimization dynamic, not curated sample quality.
  • The Necessity of Fast Inner Adaptation: Ablating the task-specific adaptation stage (running standard episodic multi-task updates directly on query batches, denoted -STA) consistently degrades performance. Local adaptation forces the outer loop to learn an initialization that is primed for fast transfer across varied tasks.

Key Results

  • Massive Data Reductions:
  • Flanv2: On LLaMA2-13B, META-LORA achieved a 46.32 BBH score using only 100 samples per task (4,300 samples total), outperforming full-data LoRA (40.9), HydraLoRA (44.7), and R-LoRA (45.1), which consumed all 325,783 examples.
  • Five-Task Setting: On LLaMA2-7B, META-LORA scored 39.00% average with just 50 examples per task (250 total), outperforming full-data LoRA (28.70%) and HydraLoRA (38.03%).
  • Multilingual Transfer: Fine-tuned on only 50 examples per language (2,600 total), META-LORA achieved an average score of 54.61% (7B) and 55.87% (13B), surpassing the official Bactrian-X models (49.99% and 53.69%) trained on 3.4 million instruction pairs.
  • Compute & Time Efficiency: On the 5-task dataset, META-LORA processed only 64,000 samples over 1,000 steps compared to 487,075 samples over ~60,885 steps for baselines. Training time for LLaMA2-7B dropped from 37 hours (HydraLoRA) and 17 hours (LoRA) to just 3 hours.

Limitations and Future Work

  • Hyperparameter Sensitivity: The ablation studies reveal that inner-loop hyperparameters require delicate calibration: setting inner steps $k > 3$ leads to overfitting on the support set, while sampling $n > 2$ tasks per iteration degrades performance due to conflicting gradients.
  • First-Order Approximation Constraints: The method relies on detaching the inner-loop computation graph to avoid second-order gradients. While essential for LLM scalability, this omits higher-order meta-gradient dynamics.
  • Sub-task Generalization Trade-offs: On specific factual domains like high school exams (EXAMS benchmark), META-LORA underperformed models trained on full datasets (23.84% vs. 29.14% on 7B), suggesting that extreme data efficiency can bottleneck tasks that demand dense, explicit factual memorization rather than transferable reasoning patterns.

Summary updated

Abstract

Low-Rank Adaptation (LoRA) has emerged as one of the most widely used parameter-efficient fine-tuning (PEFT) methods for adapting large language models (LLMs) to downstream tasks. While highly effective in single-task settings, it struggles to efficiently leverage inter-task knowledge in complex multi-task learning scenarios, often requiring substantial task-specific data to achieve optimal performance. To address this limitation, we introduce MeTA-LoRA, a two-stage optimization framework that significantly improves data efficiency in multi-task adaptation. In the first stage, task-specific LoRA adapters are learned using only a few samples from each involved dataset, enabling rapid adaptation without large-scale supervision. In the second stage, the shared LoRA adapter is updated by aggregating gradients from multiple tasks to promote knowledge transfer across tasks, further reducing data usage by leveraging common patterns. In both multi-task learning and multilingual learning scenarios, our method matches or surpasses the performance of traditional full-data LoRA fine-tuning approaches, while using significantly less task-specific data.
BibTeX
@misc{cheng2025meta,
  author = {Bo Cheng and Xu Wang and Jinda Liu and Yuan Wu and Yi Chang},
  title = {{MeTA-LoRA: Data-Efficient Multi-Task Fine-Tuning for Large Language Models}},
  year = {2025},
  eprint = {2510.11598},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.11598},
}

Loading related content...

Figure