MoKA: mixture of Kronecker adapters

Mohammadreza Sadeghi, Mahsa Ghazvini Nejad, Hamed Jafarzadeh Asl, Yu Gu, Yuanhao Yu, Masoud Asgharian, Vahid Partovi Nia

Introduction

Fine-tuning Large Language Models (LLMs) requires balancing performance with resource constraints. While Low-Rank Adaptation (LoRA) is the standard for Parameter-Efficient Fine-Tuning (PEFT), its low-rank constraint can limit the model's ability to capture complex patterns. Researchers from Huawei’s Noah’s Ark Lab and McGill University have introduced MoKA (Mixture of Kronecker Adapters), a method that replaces simple low-rank matrices with an expressive mixture of Kronecker products. MoKA achieves state-of-the-art performance on instruction-tuning and reasoning tasks while using up to 27x fewer trainable parameters than existing methods like QLoRA.

The Rank Bottleneck in PEFT

Standard PEFT methods like LoRA approximate weight updates ($\Delta W$) using two small matrices ($A$ and $B$) such that $\Delta W = AB$. This forces the update to be low-rank. While efficient, this constraint limits expressiveness; if a task requires a high-rank update to capture complex logic, LoRA may struggle.

Kronecker products offer a solution by representing high-rank (even full-rank) matrices with very few parameters. However, previous Kronecker-based adapters suffered from rigid structures and hardware inefficiency on GPUs optimized for dense matrix multiplication. MoKA addresses these issues by making Kronecker adapters flexible, gated, and hardware-friendly.

Mixture of Kronecker Adapters (MoKA)

The core of MoKA is a weighted sum of $r$ Kronecker factors: $$\Delta W = \sum_{i=1}^{r} \alpha_i (A_i \otimes B_i)$$

Detailed architecture of MoKA, illustrating the reshaping trick and gating mechanism.

Key Innovations

  • Gating Mechanism: Learnable softmax gating weights ($\alpha_i$) allow the model to dynamically prioritize different Kronecker "shapes" based on input context.
  • Diverse Filter Shapes: MoKA employs various shapes (e.g., $64 \times 64$, $16 \times 256$) to capture different structural patterns in the weights.
  • Hardware Efficiency: To overcome GPU limitations, MoKA uses a "Matrix-Product Trick," reformulating the Kronecker-vector product into standard matrix operations: $(A \otimes B)x = \text{vec}(B \cdot \text{reshape}(x) \cdot A^T)$.
  • MoKA$_s$: A hyper-efficient variant where one factor is fixed as an identity matrix, creating a mixture of block-diagonal adapters.

Experimental Evaluation

MoKA was evaluated using 4-bit quantized versions of LLaMA2-7B and LLaMA3-8B across two domains:

  1. Instruction-Tuning: Evaluated on the MMLU benchmark across seven datasets (e.g., Alpaca, Guanaco).
  2. Commonsense Reasoning: Zero-shot evaluation on eight tasks including BoolQ, HellaSwag, and ARC.

Results and Insights

MoKA consistently outperformed stronger baselines like QLoRA and QDoRA while being significantly smaller.

Performance comparison on instruction-tuning tasks showing MoKA's parameter efficiency.
  • Parameter Efficiency: On LLaMA2-7B, MoKA used only 5.2M trainable parameters compared to QLoRA’s 62.2M (a ~12x reduction) while achieving higher average accuracy on MMLU (43.16% vs 36.46%). On LLaMA3-8B, MoKA achieved better results than QLoRA with 14x fewer parameters.
  • Reasoning Performance: In commonsense reasoning, 4-bit MoKA even outperformed the original 16-bit "no fine-tuning" baseline, effectively recovering performance lost during quantization.
  • The Power of Gating: Ablation studies confirmed that the gating mechanism is crucial; learning which "filter shape" to prioritize for a specific input significantly improves performance over simple averaging.

Conclusion

MoKA breaks the rank bottleneck of traditional PEFT by leveraging the high-rank capacity of Kronecker products within a flexible, gated framework. While current shapes are pre-defined, future work may explore automated shape selection or deeper integration with advanced quantization techniques. MoKA provides a mathematically grounded and empirically superior alternative for efficient LLM adaptation.

Summary updated

Abstract

Parameter-efficient fine-tuning (PEFT) is essential for reducing the computational overhead of large language models (LLMs). Low-rank family adapters are commonly used to control the parameter size efficiently while maintaining the generative power of LLMs. However, their limited expressiveness due to the rank constraint often restricts their performance on complex tasks. We propose Mixture of Kronecker Adapters (MoKA), a new generation of Kronecker adapters that addresses this limitation by modeling weight updates as a mixture of Kronecker products. Our proposed adapter leverages a gating mechanism that measures the importance of each Kronecker factor, enabling more expressive adaptation. Moreover, MoKA enables a rank flexibility that provides a better trade-off between parameter efficiency and accuracy. To ensure hardware efficiency, we reformulate Kronecker computations using standard matrix operations, allowing seamless deployment on GPU-optimized hardware. We conduct extensive experiments on instruction-tuning and commonsense reasoning tasks using low-bit quantized versions of LLaMA2-7B and LLaMA3-8B models. MoKA not only outperforms PEFT baselines, but also reduces the number of trainable parameters up to 27x, achieving state-of-the-art trade-offs between performance and parameter efficiency.
BibTeX
@misc{sadeghi2025moka,
  author = {Mohammadreza Sadeghi and Mahsa Ghazvini Nejad and Hamed Jafarzadeh Asl and Yu Gu and YUANHAO YU and Masoud Asgharian and Vahid Partovi Nia},
  title = {{MoKA: mixture of Kronecker adapters}},
  year = {2025},
  eprint = {2508.03527},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.03527},
}
Figure