SPARTA: An Optimization Framework for Differentially Private Sparse Fine-Tuning
Name
3711896.3736842.pdf
Size
1.22 MB
Format
Adobe PDF
Checksum (MD5)
e179a1d0ae814308be6e821948f86626
Author(s)
Makni, Mehdi
Behdin, Kayhan
Afriat, Gabriel
Xu, Zheng
Vassilvitskii, Sergei
Ponomareva, Natalia
Mazumder, Rahul
Hazimeh, Hussein
Date Issued
August 3, 2025
Publisher
ACM|Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2
Citation
Mehdi Makni, Kayhan Behdin, Gabriel Afriat, Zheng Xu, Sergei Vassilvitskii, Natalia Ponomareva, Rahul Mazumder, and Hussein Hazimeh. 2025. SPARTA: An Optimization Framework for Differentially Private Sparse Fine-Tuning. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2 (KDD '25). Association for Computing Machinery, New York, NY, USA, 2090–2101.
Version
Final published version
Abstract
Differentially private stochastic gradient descent (DP-SGD) is broadly considered to be the gold standard for training and fine-tuning neural networks under differential privacy (DP). With the increasing availability of high-quality pre-trained model checkpoints (e.g., vision and language models), fine-tuning has become a popular strategy. However, despite recent progress in understanding and applying DP-SGD for private transfer learning tasks, significant challenges remain - most notably, the performance gap between models fine-tuned with DP-SGD and their non-private counterparts. Sparse fine-tuning on private data has emerged as an alternative to full-model fine-tuning -- recent work has shown that privately fine-tuning only a small subset of model weights and keeping the rest of the weights fixed can lead to better performance. In this work, we propose a new approach for sparse fine-tuning of neural networks under DP. Existing work on private sparse finetuning often used fixed choice of trainable weights (e.g., updating only the last layer), or relied on public model's weights to choose the subset of weights to modify. Such choice of weights remains suboptimal. In contrast, we explore an optimization-based approach, where our selection method makes use of the private gradient information, while using off the shelf privacy accounting techniques. Our numerical experiments on several computer vision models and datasets show that our parameter selection method leads to better prediction accuracy, compared to full-model private fine-tuning or existing private sparse fine-tuning approaches. Our code is available here: https://github.com/mazumder-lab/SPARTA/tree/main
Description
KDD ’25, Toronto, ON, Canada
MIT Department
Massachusetts Institute of Technology. Operations Research Center
Sloan School of Management
Terms of Use
Creative Commons Attribution
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1145/3711896.3736842
