收藏切换
A Fast and Secure Transformer Inference Scheme with Secure Multi-Party Computation
收藏切换
Weixin Liu1, Yewei Guan1, Jiarong Huo1, Yuanchao Ding1, Hua Guo1, 2, *, Bo Li2
Journal of Computer Research and Development | 2024, 61(5) : 1218 - 1229
Less
收藏切换
Journal of Computer Research and Development | 2024, 61(5): 1218-1229
• Special Issue on Large Models and Safety •
A Fast and Secure Transformer Inference Scheme with Secure Multi-Party Computation
Full
Weixin Liu1, Yewei Guan1, Jiarong Huo1, Yuanchao Ding1, Hua Guo1, 2, *, Bo Li2
Affiliations
  • 1School of Cyber Science and Technology, Beihang University, Beijing 100191
  • 2State Key Laboratory of Complex & Critical Software Environment (Beihang University), Beijing 100878
  • born in 2001. Master candidate. His main research interests include privacy-preserving machine learning, cryptography

    born in 1999. PhD candidate. His main research interests include secure multi-party computation and privacy-preserving machine learning

    born in 2002. Undergraduate. His main research interest includes applied cryptography

    born in 1999. Master candidate. His main research interests include privacy-preserving machine learning and cryptography

    born in 1980. PhD, associate professor. Member of CCF. Her main research interests include privacy-preserving machine learning and cryptography

    born in 1981. PhD, associate professor. Member of CCF. His main research interests include privacy-preserving machine learning and network security

About Author:

born in 2001. Master candidate. His main research interests include privacy-preserving machine learning, cryptography

born in 1999. PhD candidate. His main research interests include secure multi-party computation and privacy-preserving machine learning

born in 2002. Undergraduate. His main research interest includes applied cryptography

born in 1999. Master candidate. His main research interests include privacy-preserving machine learning and cryptography

born in 1980. PhD, associate professor. Member of CCF. Her main research interests include privacy-preserving machine learning and cryptography

born in 1981. PhD, associate professor. Member of CCF. His main research interests include privacy-preserving machine learning and network security

Published: 2024-05-01 doi: 10.7544/issn1000-1239.202330966
Outline
收藏切换

Transformer has been widely used in many fields such as natural language processing and computer vision, and has outstanding performance. The users’ data will be leaked to the Transformer model provider during inference. With the increasing public attention on data privacy, the above data leakage problem has triggered researchers’ study on secure Transformer inference. Implementing secure Transformer inference with secure multi-party computation (MPC) is today’s hot topic. Due to the widely existence of non-linear functions in Transformer, it is hard to use MPC to implement secure Transformer inference, which leads to huge computation and communication cost. We focus on Softmax attention, bottleneck in secure Transformer inference, and propose two kinds of MPC-friendly attention mechanism, Softmax freeDiv Attention and 2Quad freeDiv Attention. By replacing the Softmax attention in Transformer with the MPC-friendly attention mechanism proposed, combining with the replacement of activation function GeLU and knowledge distillation, we propose an MPC-friendly Transformer convert framework, which can convert Transformer model to an MPC-friendly one, so as to improve the performance of secure Transformer inference later. Based on the proposed MPC-friendly Transformer convert framework , we perform secure Bert-Base inference on SST-2 in the LAN setting, using privacy computing protocols provided by secure processing unit (SPU). The result shows that the secure inference achieves 2.26 times speedup while maintaining the accuracy with non-approximation model.

secure inference  /  Transformer  /  secure multi-party computation (MPC)  /  secure processing unit (SPU)  /  knowledge distillation
刘伟欣, 管晔玮, 霍嘉荣, 丁元朝, 郭华, 李博. 一种基于安全多方计算的快速Transformer安全推理方案. 计算机研究与发展, 2024 , 61 (5) : 1218 -1229 . DOI: 10.7544/issn1000-1239.202330966
Weixin Liu, Yewei Guan, Jiarong Huo, Yuanchao Ding, Hua Guo, Bo Li. A Fast and Secure Transformer Inference Scheme with Secure Multi-Party Computation[J]. Journal of Computer Research and Development, 2024 , 61 (5) : 1218 -1229 . DOI: 10.7544/issn1000-1239.202330966
Year 2024 volume 61 Issue 5
75
34
Cite this Article
BibTeX
Article Info
doi: 10.7544/issn1000-1239.202330966
  • Receive Date:2023-12-01
  • Online Date:2026-11-17
  • Published:2024-05-01
Article Data
Affiliations
History
  • Received:2023-12-01
  • Revised:2024-03-11
  • Accepted:2024-03-11
Affiliations
    1School of Cyber Science and Technology, Beihang University, Beijing 100191
    2State Key Laboratory of Complex & Critical Software Environment (Beihang University), Beijing 100878
References
Share
https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202330966
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT