xsci-prod-verify-2308.06522

Counts differ because each database indexes a different set of publications. We treat OpenAlex as the canonical count; Google Scholar is not shown (no API, and crawling it violates its ToS).

Transfer learning via fine-tuning pre-trained transformer models has gained significant suc- cess in delivering state-of-the-art results across various NLP tasks. In the absence of central- ized data, Federated Learning (FL) can benefit from distributed and private data of the FL edge clients for fine-tuning. However, due to the limited communication, computation, and stor- age capabilities of edge devices and the huge sizes of popular transformer models, efficient fine-tuning is crucial to make federated training feasible. This work explores the opportunities and challenges associated with applying param- eter efficient fine-tuning (PEFT) methods in different FL settings for language tasks. Specif- ically, our investigation reveals that as the data across users becomes more diverse, the gap between fully fine-tuning the model and em- ploying PEFT methods widens. To bridge this performance gap, we propose a method called SLoRA, which overcomes the key limitations of LoRA in high heterogeneous data scenarios through a novel data-driven initialization tech- nique. Our experimental results demonstrate that SLoRA achieves performance comparable to full fine-tuning, with significant sparse up- dates with approximately∼1% density while reducing training time by up to 90%.