# AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning

## Origin paper

**Authors**: Qingru Zhang, Minshuo Chen, A. Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, Tuo Zhao  
**Year**: 2023

## Abstract

Fine-tuning large pre-trained language models on downstream tasks has become an important paradigm in NLP. However, common practice fine-tunes all of the parameters in a pre-trained model, which becomes prohibitive when a large number of downstream tasks are present. Therefore, many fine-tuning methods are proposed to learn incremental updates of pre-trained weights in a parameter efficient way, e.g., low-rank increments. These methods often evenly distribute the budget of incremental updates across all pre-trained weight matrices, and overlook the varying importance of different weight parameters. As a consequence, the fine-tuning performance is suboptimal. To bridge this gap, we propose AdaLoRA, which adaptively allocates the parameter budget among weight matrices according to their importance score. In particular, AdaLoRA parameterizes the incremental updates in the form of singular value decomposition. Such a novel approach allows us to effectively prune the singular values of unimportant updates, which is essentially to reduce their parameter budget but circumvent intensive exact SVD computations. We conduct extensive experiments with several pre-trained models on natural language processing, question answering, and natural language generation to validate the effectiveness of AdaLoRA. Results demonstrate that AdaLoRA manifests notable improvement over baselines, especially in the low budget settings. Our code is publicly available at [GitHub](https://github.com/QingruZhang/AdaLoRA).

## Related Works

The following is a list of derivative works relevant to the topic:

1. **Title**: Bayesian-LoRA: LoRA based Parameter Efficient Fine-Tuning using Optimal Quantization levels and Rank Values through Differentiable Bayesian Gates  
   **Authors**: Cristian Meo, Ksenia Sycheva, Anirudh Goyal, Justin Dauwels  
   **Year**: 2024

2. **Title**: DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distribution  
   **Authors**: Yulong Mao, Kaiyu Huang, Changhao Guan, Ganglin Bao, Fengran Mo, Jinan Xu  
   **Year**: 2024

3. **Title**: Towards a Unified View of Parameter-Efficient Transfer Learning  
   **Authors**: Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, Graham Neubig  
   **Year**: 2021

4. **Title**: PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptation  
   **Authors**: Nadav Benedek, Lior Wolf  
   **Year**: 2024

5. **Title**: ElaLoRA: Elastic & Learnable Low-Rank Adaptation for Efficient Model Fine-Tuning  
   **Authors**: Huandong Chang, Zicheng Ma, Mingyuan Ma, Zhenting Qi, Andrew Sabot, Hongbo Jiang, H. Kung  
   **Year**: 2025

6. **Title**: Delta-LoRA: Fine-Tuning High-Rank Parameters with the Delta of Low-Rank Matrices  
   **Authors**: Bojia Zi, Xianbiao Qi, Lingzhi Wang, Jianan Wang, Kam-Fai Wong, Lei Zhang  
   **Year**: 2023

7. **Title**: Parameter-Efficient Fine-Tuning Design Spaces  
   **Authors**: Jiaao Chen, Aston Zhang, Xingjian Shi, Mu Li, Alexander J. Smola, Diyi Yang  
   **Year**: 2023

8. **Title**: Sparse Low-rank Adaptation of Pre-trained Language Models  
   **Authors**: Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, Maosong Sun  
   **Year**: 2023

9. **Title**: IncreLoRA: Incremental Parameter Allocation Method for Parameter-Efficient Fine-tuning  
   **Authors**: Feiyu F. Zhang, Liangzhi Li, Junhao Chen, Zhouqian Jiang, Bowen Wang, Yiming Qian  
   **Year**: 2023

10. **Title**: Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learning  
    **Authors**: Clifton A. Poth, Hannah Sterz, Indraneil Paul, Sukannya Purkayastha, Leon Arne Engländer, Timo Imhof, Ivan Vuli'c, Sebastian Ruder, Iryna Gurevych, Jonas Pfeiffer  
    **Year**: 2023

### How to read the graph

- Each node is an academic paper related to the origin paper.
- Papers are arranged according to their similarity (this is not a citation tree).
- Node size is the number of citations.
- Node color is the publishing year.
- **Similar** papers have strong connecting lines and cluster together.

---
**Download**: [PDF](https://arxiv.org/pdf/2303.10512.pdf) and [Semantic Scholar](https://www.semanticscholar.org/paper/b612fc6af23cccf2133c2ea40597453ab40dc2c3).  
**Open access**.