How Module Type and Rank Impact LoRA’s Effectiveness in Model Training
Table of Links Abstract and 1 Introduction 2 Background 3 Experimental Setup and 3.1 Datasets for Continued Pretraining (CPT) and Instruction Finetuning (IFT) 3.2 Measuring Learning with Coding and Math Benchmarks (target domain evaluation) 3.3 Forgetting Metrics (source domain evaluation) 4 Results 4.1 LoRA underperforms full finetuning in programming and math tasks 4.2 LoRA forgets … Read more