Article(id=1329461181250646298, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1329460877864055544, articleNumber=null, orderNo=null, doi=10.7544/issn1000-1239.202330966, pmid=null, cstr=null, oa=null, hot=null, price=null, onlineType=0, articleFormat=0, articleType=null, articleTypeStr=null, receivedDate=1701360000000, receivedDateStr=2023-12-01, revisedDate=1710086400000, revisedDateStr=2024-03-11, acceptedDate=1710086400000, acceptedDateStr=2024-03-11, onlineDate=1794897845804, onlineDateStr=2026-11-17, pubDate=1714492800000, pubDateStr=2024-05-01, doiRegisterDate=null, doiRegisterDateStr=null, onlineIssueDate=1794897845804, onlineIssueDateStr=2026-11-17, onlineJustAcceptDate=null, onlineJustAcceptDateStr=null, onlineFirstDate=null, onlineFirstDateStr=null, sourceXml=null, magXml=null, createTime=1794897845804, creator=13701087609, updateTime=1794897845804, updator=13701087609, issue=Issue{id=1329460877864055544, tenantId=1146029695717560320, journalId=1324759914146529478, year='2024', volume='61', issue='5', pageStart='1085', pageEnd='1347', issueExtLink='null', onlineDate='null', pubDate='1714492800000', pubDateStr='2024-05-01', beforeIssueId=null, nextIssueId=null, price=null, status=1, issueComplete=1, articleOrder=1, issueType=-1, specialIssue=null, createTime=1794897773471, creator='13701087609', updateTime=1794897773471, updator='13701087609', preIssue=null, nextIssue=null, articleTotal=null, ext=null, issueFiles=null, downloadFileDto=null}, startPage=1218, endPage=1229, ext={EN=ArticleExt(id=1329461181431001371, articleId=1329461181250646298, tenantId=1146029695717560320, journalId=1324759914146529478, language=EN, title=A Fast and Secure Transformer Inference Scheme with Secure Multi-Party Computation, columnId=1329460878547727097, journalTitle=Journal of Computer Research and Development, columnName=Special Issue on Large Models and Safety, runingTitle=null, highlight=null, articleAbstract=

Transformer has been widely used in many fields such as natural language processing and computer vision, and has outstanding performance. The users’ data will be leaked to the Transformer model provider during inference. With the increasing public attention on data privacy, the above data leakage problem has triggered researchers’ study on secure Transformer inference. Implementing secure Transformer inference with secure multi-party computation (MPC) is today’s hot topic. Due to the widely existence of non-linear functions in Transformer, it is hard to use MPC to implement secure Transformer inference, which leads to huge computation and communication cost. We focus on Softmax attention, bottleneck in secure Transformer inference, and propose two kinds of MPC-friendly attention mechanism, Softmax freeDiv Attention and 2Quad freeDiv Attention. By replacing the Softmax attention in Transformer with the MPC-friendly attention mechanism proposed, combining with the replacement of activation function GeLU and knowledge distillation, we propose an MPC-friendly Transformer convert framework, which can convert Transformer model to an MPC-friendly one, so as to improve the performance of secure Transformer inference later. Based on the proposed MPC-friendly Transformer convert framework , we perform secure Bert-Base inference on SST-2 in the LAN setting, using privacy computing protocols provided by secure processing unit (SPU). The result shows that the secure inference achieves 2.26 times speedup while maintaining the accuracy with non-approximation model.

, authors=Weixin Liu1, Yewei Guan1, Jiarong Huo1, Yuanchao Ding1, Hua Guo1, 2, *, Bo Li2, authorsList=Weixin Liu, Yewei Guan, Jiarong Huo, Yuanchao Ding, Hua Guo, Bo Li, authorCompany=null, correspAuthors=Hua Guo, authorNote=

born in 2001. Master candidate. His main research interests include privacy-preserving machine learning, cryptography

born in 1999. PhD candidate. His main research interests include secure multi-party computation and privacy-preserving machine learning

born in 2002. Undergraduate. His main research interest includes applied cryptography

born in 1999. Master candidate. His main research interests include privacy-preserving machine learning and cryptography

born in 1980. PhD, associate professor. Member of CCF. Her main research interests include privacy-preserving machine learning and cryptography

born in 1981. PhD, associate professor. Member of CCF. His main research interests include privacy-preserving machine learning and network security

, correspAuthorsNote=null, copyrightStatement=Copyright © 2024 Journal of Computer Research and Development. All rights reserved., copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=null, magXml=null, pdfUrl=null, pdf=null, pdfFileSize=null, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=null, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=null, mapNumber=null, fund=null), CN=ArticleExt(id=1329461185226846519, articleId=1329461181250646298, tenantId=1146029695717560320, journalId=1324759914146529478, language=CN, title=一种基于安全多方计算的快速Transformer安全推理方案, columnId=1329460878694527739, journalTitle=计算机研究与发展, columnName=大模型与安全专题, runingTitle=null, highlight=null, articleAbstract=

Transformer模型在自然语言处理、计算机视觉等众多领域得到了广泛应用,并且有着突出的表现. 在Transformer的推理应用中用户的数据会被泄露给模型提供方. 随着数据隐私问题愈发得到公众的关注,上述数据泄露问题引发了学者们对Transformer安全推理的研究,使用安全多方计算(secure multi-party computation,MPC)实现Transformer模型的安全推理是当前的一个研究热点. 由于Transformer模型中存在大量非线性函数,因此使用MPC技术实现Transformer安全推理会造成巨大的计算和通信开销. 针对Transformer安全推理过程中开销较大的Softmax注意力机制,提出了2种MPC友好的注意力机制Softmax freeDiv Attention和2Quad freeDiv Attention. 通过将Transformer模型中的Softmax注意力机制替换为新的MPC友好的注意力机制,同时结合激活函数GeLU的替换以及知识蒸馏技术,提出了一个MPC友好的Transformer转换框架,通过将Transformer模型转化为MPC友好的Transformer模型,提高Transformer安全推理的效率. 在局域网环境下使用安全处理器(secure processing unit, SPU)提供的隐私计算协议,基于所提出的MPC友好的Transformer转换框架,在SST-2上使用Bert-Base进行安全推理. 测试结果表明,在保持推理准确率与无近似模型一致的情况下,安全推理计算效率提高2.26倍.

, authors=刘伟欣1, 管晔玮1, 霍嘉荣1, 丁元朝1, 郭华1, 2, *, 李博2, authorsList=刘伟欣, 管晔玮, 霍嘉荣, 丁元朝, 郭华, 李博, authorCompany=null, correspAuthors=郭华, authorNote=

2001年生. 硕士研究生. 主要研究方向为隐私保护机器学习、密码学

1999年生. 博士研究生. 主要研究方向为安全多方计算、隐私保护机器学习

2002年生. 本科生. 主要研究方向为应用密码学

1999年生. 硕士研究生. 主要研究方向为隐私保护机器学习、密码学

1980年生. 博士,副教授. CCF会员. 主要研究方向为隐私保护机器学习、密码学

1981年生. 博士,副教授. CCF会员. 主要研究方向为隐私保护机器学习、网络安全

, correspAuthorsNote=
郭华()
, copyrightStatement=版权所有 © 2024 计算机研究与发展编辑部, copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=LvG0NZcFC1bu6tM0bzaYXQ==, magXml=ydHEmZ9aMUPJ53QjYxza/g==, pdfUrl=null, pdf=rRhRg3QJC6O4NmYKvBpjuA==, pdfFileSize=1701343, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=dMSK+euXaN91XduyHtrh4g==, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=zCjT4ZNhYkmQHBTuFfBhrw==, mapNumber=null, fund=null)}, authors=[Author(id=1329461185600139583, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=sy2339130lwx@buaa.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329461185663054145, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, authorId=1329461185600139583, language=EN, stringName=Weixin Liu, firstName=Weixin, middleName=null, lastName=Liu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1School of Cyber Science and Technology, Beihang University, Beijing 100191, bio={"content":"

born in 2001. Master candidate. His main research interests include privacy-preserving machine learning, cryptography

"}, bioImg=null, bioContent=

born in 2001. Master candidate. His main research interests include privacy-preserving machine learning, cryptography

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329461185734357314, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, authorId=1329461185600139583, language=CN, stringName=刘伟欣, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1北京航空航天大学网络空间安全学院 北京 100191, bio={"img":"w14iN/nyXmi0b5gI7jd+Tg==","content":"

2001年生. 硕士研究生. 主要研究方向为隐私保护机器学习、密码学

"}, bioImg=w14iN/nyXmi0b5gI7jd+Tg==, bioContent=

2001年生. 硕士研究生. 主要研究方向为隐私保护机器学习、密码学

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329461185436561720, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, xref=1, ext=[AuthorCompanyExt(id=1329461185449144633, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, companyId=1329461185436561720, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1School of Cyber Science and Technology, Beihang University, Beijing 100191), AuthorCompanyExt(id=1329461185457533242, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, companyId=1329461185436561720, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1北京航空航天大学网络空间安全学院 北京 100191)])]), Author(id=1329461185814049092, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329461185876963654, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, authorId=1329461185814049092, language=EN, stringName=Yewei Guan, firstName=Yewei, middleName=null, lastName=Guan, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1School of Cyber Science and Technology, Beihang University, Beijing 100191, bio={"content":"

born in 1999. PhD candidate. His main research interests include secure multi-party computation and privacy-preserving machine learning

"}, bioImg=null, bioContent=

born in 1999. PhD candidate. His main research interests include secure multi-party computation and privacy-preserving machine learning

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329461185948266823, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, authorId=1329461185814049092, language=CN, stringName=管晔玮, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1北京航空航天大学网络空间安全学院 北京 100191, bio={"img":"n8eryvL6alcGtG8ngVuApA==","content":"

1999年生. 博士研究生. 主要研究方向为安全多方计算、隐私保护机器学习

"}, bioImg=n8eryvL6alcGtG8ngVuApA==, bioContent=

1999年生. 博士研究生. 主要研究方向为安全多方计算、隐私保护机器学习

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329461185436561720, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, xref=1, ext=[AuthorCompanyExt(id=1329461185449144633, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, companyId=1329461185436561720, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1School of Cyber Science and Technology, Beihang University, Beijing 100191), AuthorCompanyExt(id=1329461185457533242, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, companyId=1329461185436561720, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1北京航空航天大学网络空间安全学院 北京 100191)])]), Author(id=1329461186019569993, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329461186090873163, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, authorId=1329461186019569993, language=EN, stringName=Jiarong Huo, firstName=Jiarong, middleName=null, lastName=Huo, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1School of Cyber Science and Technology, Beihang University, Beijing 100191, bio={"content":"

born in 2002. Undergraduate. His main research interest includes applied cryptography

"}, bioImg=null, bioContent=

born in 2002. Undergraduate. His main research interest includes applied cryptography

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329461186153787724, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, authorId=1329461186019569993, language=CN, stringName=霍嘉荣, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1北京航空航天大学网络空间安全学院 北京 100191, bio={"img":"OK3mx3xIMhIvXPEwJOiMsA==","content":"

2002年生. 本科生. 主要研究方向为应用密码学

"}, bioImg=OK3mx3xIMhIvXPEwJOiMsA==, bioContent=

2002年生. 本科生. 主要研究方向为应用密码学

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329461185436561720, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, xref=1, ext=[AuthorCompanyExt(id=1329461185449144633, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, companyId=1329461185436561720, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1School of Cyber Science and Technology, Beihang University, Beijing 100191), AuthorCompanyExt(id=1329461185457533242, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, companyId=1329461185436561720, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1北京航空航天大学网络空间安全学院 北京 100191)])]), Author(id=1329461186233479502, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, orderNo=3, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329461186313171280, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, authorId=1329461186233479502, language=EN, stringName=Yuanchao Ding, firstName=Yuanchao, middleName=null, lastName=Ding, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1School of Cyber Science and Technology, Beihang University, Beijing 100191, bio={"content":"

born in 1999. Master candidate. His main research interests include privacy-preserving machine learning and cryptography

"}, bioImg=null, bioContent=

born in 1999. Master candidate. His main research interests include privacy-preserving machine learning and cryptography

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329461186380280145, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, authorId=1329461186233479502, language=CN, stringName=丁元朝, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1北京航空航天大学网络空间安全学院 北京 100191, bio={"img":"LYSy2heXEILnbgGIz9MmrQ==","content":"

1999年生. 硕士研究生. 主要研究方向为隐私保护机器学习、密码学

"}, bioImg=LYSy2heXEILnbgGIz9MmrQ==, bioContent=

1999年生. 硕士研究生. 主要研究方向为隐私保护机器学习、密码学

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329461185436561720, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, xref=1, ext=[AuthorCompanyExt(id=1329461185449144633, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, companyId=1329461185436561720, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1School of Cyber Science and Technology, Beihang University, Beijing 100191), AuthorCompanyExt(id=1329461185457533242, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, companyId=1329461185436561720, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1北京航空航天大学网络空间安全学院 北京 100191)])]), Author(id=1329461186472554835, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, orderNo=4, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=hguo@buaa.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=1, authorType=1, ext={EN=AuthorExt(id=1329461186548052310, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, authorId=1329461186472554835, language=EN, stringName=Hua Guo, firstName=Hua, middleName=null, lastName=Guo, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, *, address=1School of Cyber Science and Technology, Beihang University, Beijing 100191
2State Key Laboratory of Complex & Critical Software Environment (Beihang University), Beijing 100878, bio={"content":"

born in 1980. PhD, associate professor. Member of CCF. Her main research interests include privacy-preserving machine learning and cryptography

"}, bioImg=null, bioContent=

born in 1980. PhD, associate professor. Member of CCF. Her main research interests include privacy-preserving machine learning and cryptography

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329461186623549783, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, authorId=1329461186472554835, language=CN, stringName=郭华, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, *, address=1北京航空航天大学网络空间安全学院 北京 100191
2复杂关键软件环境全国重点实验室(北京航空航天大学) 北京 100878, bio={"img":"98sFdXx5RqbSU2tgXeaOdQ==","content":"

1980年生. 博士,副教授. CCF会员. 主要研究方向为隐私保护机器学习、密码学

"}, bioImg=98sFdXx5RqbSU2tgXeaOdQ==, bioContent=

1980年生. 博士,副教授. CCF会员. 主要研究方向为隐私保护机器学习、密码学

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329461185436561720, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, xref=1, ext=[AuthorCompanyExt(id=1329461185449144633, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, companyId=1329461185436561720, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1School of Cyber Science and Technology, Beihang University, Beijing 100191), AuthorCompanyExt(id=1329461185457533242, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, companyId=1329461185436561720, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1北京航空航天大学网络空间安全学院 北京 100191)]), AuthorCompany(id=1329461185524642107, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, xref=2, ext=[AuthorCompanyExt(id=1329461185533030716, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, companyId=1329461185524642107, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2State Key Laboratory of Complex & Critical Software Environment (Beihang University), Beijing 100878), AuthorCompanyExt(id=1329461185541419325, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, companyId=1329461185524642107, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2复杂关键软件环境全国重点实验室(北京航空航天大学) 北京 100878)])]), Author(id=1329461188242551129, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, orderNo=5, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329461188372574555, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, authorId=1329461188242551129, language=EN, stringName=Bo Li, firstName=Bo, middleName=null, lastName=Li, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2State Key Laboratory of Complex & Critical Software Environment (Beihang University), Beijing 100878, bio={"content":"

born in 1981. PhD, associate professor. Member of CCF. His main research interests include privacy-preserving machine learning and network security

"}, bioImg=null, bioContent=

born in 1981. PhD, associate professor. Member of CCF. His main research interests include privacy-preserving machine learning and network security

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329461188427100508, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, authorId=1329461188242551129, language=CN, stringName=李博, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2复杂关键软件环境全国重点实验室(北京航空航天大学) 北京 100878, bio={"img":"r6ckzWpanNAIEVm/Ev6JLQ==","content":"

1981年生. 博士,副教授. CCF会员. 主要研究方向为隐私保护机器学习、网络安全

"}, bioImg=r6ckzWpanNAIEVm/Ev6JLQ==, bioContent=

1981年生. 博士,副教授. CCF会员. 主要研究方向为隐私保护机器学习、网络安全

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329461185524642107, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, xref=2, ext=[AuthorCompanyExt(id=1329461185533030716, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, companyId=1329461185524642107, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2State Key Laboratory of Complex & Critical Software Environment (Beihang University), Beijing 100878), AuthorCompanyExt(id=1329461185541419325, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, companyId=1329461185524642107, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2复杂关键软件环境全国重点实验室(北京航空航天大学) 北京 100878)])])], keywords=[Keyword(id=1329461188531958109, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, language=EN, orderNo=1, keyword=secure inference), Keyword(id=1329461188603261278, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, language=EN, orderNo=2, keyword=Transformer), Keyword(id=1329461188670370143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, language=EN, orderNo=3, keyword=secure multi-party computation (MPC)), Keyword(id=1329461188754256224, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, language=EN, orderNo=4, keyword=secure processing unit (SPU)), Keyword(id=1329461188821365089, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, language=EN, orderNo=5, keyword=knowledge distillation), Keyword(id=1329461188896862562, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, language=CN, orderNo=1, keyword=安全推理), Keyword(id=1329461188980748643, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, language=CN, orderNo=2, keyword=Transformer), Keyword(id=1329461189039468900, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, language=CN, orderNo=3, keyword=安全多方计算), Keyword(id=1329461189102383461, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, language=CN, orderNo=4, keyword=安全处理器), Keyword(id=1329461189161103718, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, language=CN, orderNo=5, keyword=知识蒸馏)], refs=[Reference(id=1329461192759816578, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=1, rfOrder=0, authorNames=null, journalName=null, refType=null, unstructuredReference=Hoffmann J, Borgeaud S, Mensch A, et al. An empirical analysis of compute-optimal large language model training[J]. Advances in Neural Information Processing Systems, 2022, 35: 30016−30030, articleTitle=null, refAbstract=null), Reference(id=1329461192822731139, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=2, rfOrder=1, authorNames=null, journalName=null, refType=null, unstructuredReference=Chan S, Santoro A, Lampinen A, et al. Data distributional properties drive emergent in-context learning in transformers[J]. Advances in Neural Information Processing Systems, 2022, 35: 18878−18891, articleTitle=null, refAbstract=null), Reference(id=1329461192881451396, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=3, rfOrder=2, authorNames=null, journalName=null, refType=null, unstructuredReference=Liu Ze, Lin Yutong, Cao Yue, et al. Swin transformer: Hierarchical vision transformer using shifted windows[C]//Proc of the IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE , 2021: 10012−10022, articleTitle=null, refAbstract=null), Reference(id=1329461192944365957, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=4, rfOrder=3, authorNames=null, journalName=null, refType=null, unstructuredReference=Liu Ze, Hu Han, Lin Yutong, et al. Swin transformer v2: Scaling up capacity and resolution[C]//Proc of the IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 12009−12019, articleTitle=null, refAbstract=null), Reference(id=1329461193019863430, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=5, rfOrder=4, authorNames=null, journalName=null, refType=null, unstructuredReference=Jawalkar N, Gupta K, Basu A, et al. Orca: FSS-based secure training with GPUs[J]. Cryptology ePrint Archive, 2023, articleTitle=null, refAbstract=null), Reference(id=1329461193082777991, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=6, rfOrder=5, authorNames=null, journalName=null, refType=null, unstructuredReference=Hao Meng, Li Hongwei, Chen Hanxiao, et al. Iron: Private inference on transformers[J]. Advances in Neural Information Processing Systems, 2022, 35: 15718−15731, articleTitle=null, refAbstract=null), Reference(id=1329461193149886856, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=7, rfOrder=6, authorNames=null, journalName=null, refType=null, unstructuredReference=Chen Tianyu, Bao Hangbo, Huang Shaohan, et al. THE-X: Privacy-preserving transformer inference with homomorphic encryption[C]//Findings of the Association for Computational Linguistics: ACL 2022. Stroudsburg, PA: ACL, 2022: 3510−3520, articleTitle=null, refAbstract=null), Reference(id=1329461193208607113, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=8, rfOrder=7, authorNames=null, journalName=null, refType=null, unstructuredReference=Zheng Mengxin, Lou Qian, Lei Jiang. Primer: Fast private transformer inference on encrypted data[J]. arXiv preprint, arXiv: 2303.13679, 2023, articleTitle=null, refAbstract=null), Reference(id=1329461193275715978, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=9, rfOrder=8, authorNames=null, journalName=null, refType=null, unstructuredReference=Gupta K, Jawalkar N, Mukherjee A, et al. Sigma: Secure gpt inference with function secret sharing[J]. Cryptology ePrint Archive, 2023, articleTitle=null, refAbstract=null), Reference(id=1329461193334436235, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=10, rfOrder=9, authorNames=null, journalName=null, refType=null, unstructuredReference=Juvekar C, Vaikuntanathan V, Chandrakasan A. GAZELLE: A low latency framework for secure neural network inference[C]//Proc of the 27th USENIX Conf on Security Symp. Berkeley, CA: USENIX Association, 2018: 1651−1669, articleTitle=null, refAbstract=null), Reference(id=1329461193388962188, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=11, rfOrder=10, authorNames=null, journalName=null, refType=null, unstructuredReference=Jiang Xiaoqian, Kim M, Lauter K, et al. Secure outsourced matrix computation and application to neural networks[C]//Proc of the 2018 ACM SIGSAC Conf on Computer and Communications Security. New York: Association for Computing Machinery, 2018: 1209−1222, articleTitle=null, refAbstract=null), Reference(id=1329461193447682445, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=12, rfOrder=11, authorNames=null, journalName=null, refType=null, unstructuredReference=Mohassel P, Zhang Y. SecureML: A system for scalable privacy-preserving machine learning[C]//Proc of 2017 IEEE Symp on Security and Privacy (SP). Piscataway, NJ: IEEE, 2017: 19−38, articleTitle=null, refAbstract=null), Reference(id=1329461193527374222, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=13, rfOrder=12, authorNames=null, journalName=null, refType=null, unstructuredReference=Huang Zhicong, Lu Wenjie, Hong Cheng, et al. Cheetah: Lean and fast secure two-party deep neural network inference[C]//Proc of the 31st USENIX Security Symp (USENIX Security 22). Berkeley, CA: USENIX Association, 2022: 809−826, articleTitle=null, refAbstract=null), Reference(id=1329461193594483087, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=14, rfOrder=13, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Ning, Xiao Xiaohui, Yang Yin, et al. Collecting and analyzing multidimensional data with local differential privacy[C]//Proc of the 2019 IEEE 35th Int Conf on Data Engineering (ICDE). Piscataway, NJ: IEEE, 2019: 638−649, articleTitle=null, refAbstract=null), Reference(id=1329461193657397648, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=15, rfOrder=14, authorNames=null, journalName=null, refType=null, unstructuredReference=Truong J B, Gallagher W, Guo Tian, et al. Memory-efficient deep learning inference in trusted execution environments[C]//Proc of the 2021 IEEE Int Conf on Cloud Engineering (IC2E). Piscataway, NJ: IEEE, 2021: 161−167, articleTitle=null, refAbstract=null), Reference(id=1329461193728700817, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=16, rfOrder=15, authorNames=null, journalName=null, refType=null, unstructuredReference=Akavia A, Leibovich M, Resheff Y S, et al. Privacy-preserving decision trees training and prediction[J]. ACM Transactions on Privacy and Security, 2022, 25(3): 1−30, articleTitle=null, refAbstract=null), Reference(id=1329461193779032466, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=17, rfOrder=16, authorNames=null, journalName=null, refType=null, unstructuredReference=Park S, Byun J, Lee J. Privacy-preserving fair learning of support vector machine with homomorphic encryption[C]//Proc of the ACM Web Conf 2022. New York: ACM, 2022: 3572−3583, articleTitle=null, refAbstract=null), Reference(id=1329461193850335635, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=18, rfOrder=17, authorNames=null, journalName=null, refType=null, unstructuredReference=Mohassel P, Rindal P. ABY3: A mixed protocol framework for machine learning[C]//Proc of the 2018 ACM SIGSAC Conf on Computer and Communications Security. New York: ACM, 2018: 35−52, articleTitle=null, refAbstract=null), Reference(id=1329461193930027412, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=19, rfOrder=18, authorNames=null, journalName=null, refType=null, unstructuredReference=Rathee D, Rathee M, Kumar N, et al. Cryptflow2: Practical 2-party secure inference[C]//Proc of the 2020 ACM SIGSAC Conf on Computer and Communications Security. New York: ACM, 2020: 325−342, articleTitle=null, refAbstract=null), Reference(id=1329461193992941973, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=20, rfOrder=19, authorNames=null, journalName=null, refType=null, unstructuredReference=Hou Xiaoyang, Liu Jian, Li Jingyu, et al. Ciphergpt: Secure two-party gpt inference[J]. Cryptology ePrint Archive, 2023, articleTitle=null, refAbstract=null), Reference(id=1329461194064245142, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=21, rfOrder=20, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Dacheng, Shao Rulin, Wang Hongyi, et al. MPCFormer: Fast, performant and private transformer inference with MPC[J]. arXiv preprint, arXiv: 2211.01452, 2022, articleTitle=null, refAbstract=null), Reference(id=1329461194131354007, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=22, rfOrder=21, authorNames=null, journalName=null, refType=null, unstructuredReference=Zeng Wenxuan, Li Meng, Xiong Wenjie, et al. MPCViT: Searching for MPC-friendly vision transformer with heterogeneous attention[J]. arXiv preprint, arXiv: 2211.13955, 2022, articleTitle=null, refAbstract=null), Reference(id=1329461194194268568, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=23, rfOrder=22, authorNames=null, journalName=null, refType=null, unstructuredReference=Mishra P, Lehmkuhl R, Srinivasan A, et al. Delphi: A cryptographic inference system for neural networks[C]//Proc of the 29th USENIX Conf on Security Symp. Berkeley, CA: USENIX Association, 2020: 2505−2522, articleTitle=null, refAbstract=null), Reference(id=1329461194257183129, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=24, rfOrder=23, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Xiaolong, Girshick R, Gupta A, et al. Non-local neural networks[C]//Proc of the IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2018: 7794−7803, articleTitle=null, refAbstract=null), Reference(id=1329461194336874906, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=25, rfOrder=24, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Sinong, Li B Z, Khabsa M, et al. Linformer: Self-attention with linear complexity[J]. arXiv preprint, arXiv: 2006.04768, 2020, articleTitle=null, refAbstract=null), Reference(id=1329461194403983771, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=26, rfOrder=25, authorNames=null, journalName=null, refType=null, unstructuredReference=Rathee D, Rathee M, Goli R K K, et al. Sirnn: A math library for secure RNN inference[C]//Proc of 2021 IEEE Symp on Security and Privacy (SP). Piscataway, NJ: IEEE, 2021: 1003−1020, articleTitle=null, refAbstract=null), Reference(id=1329461194462704028, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=27, rfOrder=26, authorNames=null, journalName=null, refType=null, unstructuredReference=Dong Ye, Lu Wenjie, Zheng Yancheng, et al. Puma: Secure inference of LLaMA-7B in five minutes[J]. arXiv preprint, arXiv: 2307.12533, 2023, articleTitle=null, refAbstract=null), Reference(id=1329461194546590110, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=28, rfOrder=27, authorNames=null, journalName=null, refType=null, unstructuredReference=Akimoto Y, Fukuchi K, Akimoto Y, et al. Privformer: Privacy-preserving transformer with MPC[C]//Proc of 2023 IEEE 8th European Symp on Security and Privacy (EuroS&P). Piscataway, NJ: IEEE, 2023: 392−410, articleTitle=null, refAbstract=null), Reference(id=1329461194613698975, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=29, rfOrder=28, authorNames=null, journalName=null, refType=null, unstructuredReference=Wagh S, Tople S, Benhamouda F, et al. Falcon: Honest-majority maliciously secure framework for private deep learning[J]. arXiv preprint, arXiv: 2004.02229, 2020, articleTitle=null, refAbstract=null), Reference(id=1329461194672419232, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=30, rfOrder=29, authorNames=null, journalName=null, refType=null, unstructuredReference=Chou E, Beal J, Levy D, et al. Faster Cryptonets: Leveraging sparsity for real-world encrypted inference[J]. arXiv preprint, arXiv: 1811.09953, 2018, articleTitle=null, refAbstract=null), Reference(id=1329461194739528097, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=31, rfOrder=30, authorNames=null, journalName=null, refType=null, unstructuredReference=Hinton G, Vinyals O, Dean J. Distilling the knowledge in a neural network[J]. arXiv preprint , arXiv: 1503.02531, 2015, articleTitle=null, refAbstract=null), Reference(id=1329461194798248354, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=32, rfOrder=31, authorNames=null, journalName=null, refType=null, unstructuredReference=Ma Junming, Zheng Yancheng, Feng Jun, et al. SecretFlow-SPU: A performant and user-friendly framework for privacy-preserving machine learning[C]//Proc of 2023 USENIX Annual Technical Conf (USENIX ATC 23). Berkeley, CA: USENIX Associatioin, 2023: 17−33, articleTitle=null, refAbstract=null)], funds=null, companyList=[AuthorCompany(id=1329461185436561720, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, xref=1, ext=[AuthorCompanyExt(id=1329461185449144633, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, companyId=1329461185436561720, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1School of Cyber Science and Technology, Beihang University, Beijing 100191), AuthorCompanyExt(id=1329461185457533242, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, companyId=1329461185436561720, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1北京航空航天大学网络空间安全学院 北京 100191)]), AuthorCompany(id=1329461185524642107, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, xref=2, ext=[AuthorCompanyExt(id=1329461185533030716, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, companyId=1329461185524642107, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2State Key Laboratory of Complex & Critical Software Environment (Beihang University), Beijing 100878), AuthorCompanyExt(id=1329461185541419325, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, companyId=1329461185524642107, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2复杂关键软件环境全国重点实验室(北京航空航天大学) 北京 100878)])], figs=[ArticleFig(id=1329461189274349927, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, language=EN, label=Figure 1, caption=Process of training and using ChatGPT, figureFileSmall=a52etatokTakyDFCvXFl8g==, figureFileBig=dMSK+euXaN91XduyHtrh4g==, tableContent=null), ArticleFig(id=1329461189345653096, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, language=CN, label=图1, caption=ChatGPT训练及使用过程, figureFileSmall=a52etatokTakyDFCvXFl8g==, figureFileBig=dMSK+euXaN91XduyHtrh4g==, tableContent=null), ArticleFig(id=1329461189538591081, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, language=EN, label=Figure 2, caption=Architecture of Encoder-only Transformer, figureFileSmall=jBHRFP17EYpvYLaOr2borw==, figureFileBig=+8VPwFrEUYikuloxbRC2Uw==, tableContent=null), ArticleFig(id=1329461189614088554, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, language=CN, label=图2, caption=Encoder-only Transformer 结构, figureFileSmall=jBHRFP17EYpvYLaOr2borw==, figureFileBig=+8VPwFrEUYikuloxbRC2Uw==, tableContent=null), ArticleFig(id=1329461189672808811, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, language=EN, label=Figure 3, caption=Secure Transformer inference framework, figureFileSmall=wiy/CoevWdO+dfmhdGzgCA==, figureFileBig=c8Lp4pI+KT5Z+HG9O2BPtw==, tableContent=null), ArticleFig(id=1329461189735723372, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, language=CN, label=图3, caption=Transformer安全推理框架, figureFileSmall=wiy/CoevWdO+dfmhdGzgCA==, figureFileBig=c8Lp4pI+KT5Z+HG9O2BPtw==, tableContent=null), ArticleFig(id=1329461189798637933, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, language=EN, label=Figure 4, caption=MPC-friendly Transformer convert framework, figureFileSmall=oXh5s48YT6duDUaJez78FA==, figureFileBig=35dgymF7Pso6Db8MehzxaA==, tableContent=null), ArticleFig(id=1329461189857358190, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, language=CN, label=图4, caption=MPC友好的Transformer转换框架, figureFileSmall=oXh5s48YT6duDUaJez78FA==, figureFileBig=35dgymF7Pso6Db8MehzxaA==, tableContent=null), ArticleFig(id=1329461189928661359, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, language=EN, label=Figure 5, caption=Scattergram of the mean of denominator in Softmax attention mechanism, figureFileSmall=YwgM3pJTRZOVyeZSMFqxjQ==, figureFileBig=g5CXXLGb/QFVTw1UhyMEnA==, tableContent=null), ArticleFig(id=1329461189999964528, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, language=CN, label=图5, caption=Softmax注意力机制中分母均值的散点图, figureFileSmall=YwgM3pJTRZOVyeZSMFqxjQ==, figureFileBig=g5CXXLGb/QFVTw1UhyMEnA==, tableContent=null), ArticleFig(id=1329461190075462001, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, language=EN, label=Figure 6, caption=Function fitting of the mean of denominator in Softmax attention mechanism, figureFileSmall=+uNrUrnA0VNQMOw7nAqowQ==, figureFileBig=yTTu5RlSHDwn2eeOTl8Zgg==, tableContent=null), ArticleFig(id=1329461190138376562, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, language=CN, label=图6, caption=Softmax注意力机制中分母均值的函数拟合, figureFileSmall=+uNrUrnA0VNQMOw7nAqowQ==, figureFileBig=yTTu5RlSHDwn2eeOTl8Zgg==, tableContent=null), ArticleFig(id=1329461190209679731, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, language=EN, label=Figure 7, caption=Function fitting of the mean of denominator in 2Quad attention mechanism, figureFileSmall=Z/srEc3KajRu2YiDyAqThA==, figureFileBig=hwjZb7U/AAA+V0psjAaS9Q==, tableContent=null), ArticleFig(id=1329461190272594292, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, language=CN, label=图7, caption=2Quad 注意力机制中分母均值的函数拟合, figureFileSmall=Z/srEc3KajRu2YiDyAqThA==, figureFileBig=hwjZb7U/AAA+V0psjAaS9Q==, tableContent=null), ArticleFig(id=1329461190343897461, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, language=EN, label=Table 1, caption=

Secure Computational Latency of Different Attention Mechanisms

, figureFileSmall=null, figureFileBig=null, tableContent=
注意力机制延迟/s速度提升倍数
Softmax Attention0.2361
2ReLU Attention0.1591.48
2Quad Attention0.1571.50
Scaling Attention0.0603.93
Softmax freeDiv Attention(本文)0.1531.54
2Quad freeDiv Attention(本文)0.0643.69
), ArticleFig(id=1329461190419394934, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, language=CN, label=表1, caption=

不同的注意力机制的安全计算延迟

, figureFileSmall=null, figureFileBig=null, tableContent=
注意力机制延迟/s速度提升倍数
Softmax Attention0.2361
2ReLU Attention0.1591.48
2Quad Attention0.1571.50
Scaling Attention0.0603.93
Softmax freeDiv Attention(本文)0.1531.54
2Quad freeDiv Attention(本文)0.0643.69
), ArticleFig(id=1329461190482309495, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, language=EN, label=Table 2, caption=

Performance of Different Function Replacement Schemes Compared with the Benchmark

, figureFileSmall=null, figureFileBig=null, tableContent=
注意力机制激活函数速度提升倍数准确率/%
Softmax
Attention
GeLU191.5
ReLU1.3692.0
Quad1.4292.0
2ReLU
Attention
GeLU1.1891.7
ReLU1.7591.5
Quad1.7890.9
2Quad
Attention
GeLU1.291.6
ReLU1.7991.5
Quad1.8491.8
Scaling
Attention
GeLU1.3850.9
ReLU2.1750.9
Quad2.2650.9
Softmax freeDiv
Attention(本文方法)
GeLU1.1791.7
ReLU1.5991.7
Quad1.6392.1
2Quad freeDiv
Attention(本文方法)
GeLU1.3791.6
ReLU2.1691.7
Quad2.2691.8
), ArticleFig(id=1329461190545224056, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, language=CN, label=表2, caption=

不同函数替换方案相比于基准方案的表现

, figureFileSmall=null, figureFileBig=null, tableContent=
注意力机制激活函数速度提升倍数准确率/%
Softmax
Attention
GeLU191.5
ReLU1.3692.0
Quad1.4292.0
2ReLU
Attention
GeLU1.1891.7
ReLU1.7591.5
Quad1.7890.9
2Quad
Attention
GeLU1.291.6
ReLU1.7991.5
Quad1.8491.8
Scaling
Attention
GeLU1.3850.9
ReLU2.1750.9
Quad2.2650.9
Softmax freeDiv
Attention(本文方法)
GeLU1.1791.7
ReLU1.5991.7
Quad1.6392.1
2Quad freeDiv
Attention(本文方法)
GeLU1.3791.6
ReLU2.1691.7
Quad2.2691.8
), ArticleFig(id=1329461190616527225, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, language=EN, label=Table 3, caption=

Performance of the Converted Model Compared with the Original Model

, figureFileSmall=null, figureFileBig=null, tableContent=
模型速度提升倍数准确率/%
原模型191.5
转换后的MPC友好的模型2.2691.8
), ArticleFig(id=1329461190679441786, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, language=CN, label=表3, caption=

转换后的模型与原模型的性能对比

, figureFileSmall=null, figureFileBig=null, tableContent=
模型速度提升倍数准确率/%
原模型191.5
转换后的MPC友好的模型2.2691.8
), ArticleFig(id=1329461190750744955, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, language=EN, label=Table 4, caption=

Performance of Different Function Replacement Schemes in Different Tasks

, figureFileSmall=null, figureFileBig=null, tableContent=
替换方案任务蒸馏前的
评估指标
/%
蒸馏后的
评估指标
/%
Softmax Attention
+GeLU
SST-291.5
STS-B88.7
RTE63.2
Softmax freeDiv Attention
+GeLU(本文方法)
SST-291.391.6
STS-B84.089.0
RTE55.661.4
Softmax Attention
+Quad
SST-250.992.0
STS-B4.289.2
RTE52.765.3
Softmax freeDiv Attention
+Quad(本文方法)
SST-250.991.5
STS-B4.288.0
RTE52.747.3
2Quad Attention
+Quad
SST-250.991.8
STS-B−5.784.5
RTE52.759.2
2Quad freeDiv Attention
+Quad(本文方法)
SST-249.291.5
STS-B2.981.0
RTE52.054.9
), ArticleFig(id=1329461190830436732, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, language=CN, label=表4, caption=

不同函数替换方案在不同任务中的表现

, figureFileSmall=null, figureFileBig=null, tableContent=
替换方案任务蒸馏前的
评估指标
/%
蒸馏后的
评估指标
/%
Softmax Attention
+GeLU
SST-291.5
STS-B88.7
RTE63.2
Softmax freeDiv Attention
+GeLU(本文方法)
SST-291.391.6
STS-B84.089.0
RTE55.661.4
Softmax Attention
+Quad
SST-250.992.0
STS-B4.289.2
RTE52.765.3
Softmax freeDiv Attention
+Quad(本文方法)
SST-250.991.5
STS-B4.288.0
RTE52.747.3
2Quad Attention
+Quad
SST-250.991.8
STS-B−5.784.5
RTE52.759.2
2Quad freeDiv Attention
+Quad(本文方法)
SST-249.291.5
STS-B2.981.0
RTE52.054.9
), ArticleFig(id=1329461190897545597, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, language=EN, label=Table 5, caption=

Time Consuming of MPC-friendly Transformer Convert Framework for Different Tasks

, figureFileSmall=null, figureFileBig=null, tableContent=
任务(数据集大小)耗时/min
SST-2 (67000)338
STS-B (57000)200
RTE (25000)46
), ArticleFig(id=1329461190977237374, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, language=CN, label=表5, caption=

MPC友好的Transformer转换框架针对不同任务消耗的时间

, figureFileSmall=null, figureFileBig=null, tableContent=
任务(数据集大小)耗时/min
SST-2 (67000)338
STS-B (57000)200
RTE (25000)46
), ArticleFig(id=1329461191052734847, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, language=EN, label=Table 6, caption=

Performance Comparison Between Our Work and Reference [21]

, figureFileSmall=null, figureFileBig=null, tableContent=
方案准确率/%速度提升倍数
无近似模型 Softmax Attention+GeLU91.5 (+0.0)1
2Quad Attention+Quad[21]91.8 (+0.3)1.78
2Quad freeDiv Attention+Quad(本文工作)91.8 (+0.3)2.26
), ArticleFig(id=1329461191111455105, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, language=CN, label=表6, caption=

本文工作与文献[21]的性能对比

, figureFileSmall=null, figureFileBig=null, tableContent=
方案准确率/%速度提升倍数
无近似模型 Softmax Attention+GeLU91.5 (+0.0)1
2Quad Attention+Quad[21]91.8 (+0.3)1.78
2Quad freeDiv Attention+Quad(本文工作)91.8 (+0.3)2.26
)], attaches=null, journal=Journal(id=1324759606947315908, delFlag=0, nameCn=计算机研究与发展, nameEn=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, issn=1000-1239, eissn=null, cn=11-1777/TP, coden=null, periodic=0, language=CN, oaType=null, ccby=null, superviseOffice=null, ownerOffice=null, pubOffice=null, editorOffice=null, officeType=null, aims=null, clcCode=null, officeProv=null, officeCity=null, officeAddr=null, officeZip=null, officeEmail=null, officePhone=null, editDirector=null, officeDirector=null, officeDirectorPhone=null, officeStaffNum=null, officeEmpNum=null, coverPicUrl=k2l8zgqwstuxxevFX2qxVA==, journalPrice=null, startedYear=null, abbrevIsoEn=Journal of Computer Research and Development, journalRemark=null, publicationField=null, createdTime=1793776903141, updatedTime=1793850698612, createdBy=18614031015, updatedBy=13701087609, firstLetterCn=J, firstLetterEn=J, subjectCode=Electronic Information, subjectName=null, subjectCodeEn=Electronic Information, subjectNameEn=null, picCn=k2l8zgqwstuxxevFX2qxVA==, picEn=r6k0Ry+RD+c/VXj5QarmDw==, jcr=null, cjcr=null, exts=[JournalExt(id=1325069127710286688, language=CN, name=计算机研究与发展, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698640, updatedTime=1793850698640, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""}), JournalExt(id=1325069127777395553, language=EN, name=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698656, updatedTime=1793850698656, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""})], databaseList=null, tenantJournalId=1324759914146529478, websiteList=[Website(id=1325070300139892771, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN, language=CN, createTime=1793850978169, createBy=18614031015, updateTime=1793851040259, updateBy=18614031015, name=计算机研究与发展-中文, tplId=1146099689490845704, title=计算机研究与发展, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070893751349360, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=articleTextType, value=kx, createTime=1793851119697, updateTime=1793851119697, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893730377837, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=banner, value=null, createTime=1793851119692, updateTime=1793851119692, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893772320883, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=grayFlag, value=0, createTime=1793851119702, updateTime=1793851119702, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893721989228, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851119690, updateTime=1793851119690, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893784903797, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=minRunFlag, value=0, createTime=1793851119705, updateTime=1793851119705, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893742960751, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic, createTime=1793851119695, updateTime=1793851119695, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893780709492, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=silenceFlag, value=0, createTime=1793851119704, updateTime=1793851119704, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893738766446, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_cn_619/, createTime=1793851119694, updateTime=1793851119694, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893755543665, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeColor, value=null, createTime=1793851119698, updateTime=1793851119698, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893763932274, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeStyle, value=null, createTime=1793851119700, updateTime=1793851119700, creator=18614031015, updator=18614031015)]), Website(id=1325070300580294698, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN, language=EN, createTime=1793850978274, createBy=18614031015, updateTime=1793851036638, updateBy=18614031015, name=计算机研究与发展-英文, tplId=1146101810881728533, title=Journal of Computer Research and Development, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070925288321148, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=articleTextType, value=kx, createTime=1793851127216, updateTime=1793851127216, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925263155321, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=banner, value=null, createTime=1793851127210, updateTime=1793851127210, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925309292671, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=grayFlag, value=0, createTime=1793851127221, updateTime=1793851127221, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925254766712, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851127208, updateTime=1793851127208, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925326069889, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=minRunFlag, value=0, createTime=1793851127225, updateTime=1793851127225, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925279932539, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic, createTime=1793851127214, updateTime=1793851127214, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925317681280, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=silenceFlag, value=0, createTime=1793851127223, updateTime=1793851127223, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925271543930, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_en_623/, createTime=1793851127212, updateTime=1793851127212, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925292515453, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeColor, value=null, createTime=1793851127217, updateTime=1793851127217, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925300904062, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeStyle, value=null, createTime=1793851127219, updateTime=1793851127219, creator=18614031015, updator=18614031015)])], journalTitle=计算机研究与发展, weixinUrl=null, journalUrl=https://crad.ict.ac.cn/, iacademicId=null, status=1, seqNo=null, journalTitleEn=Journal of Computer Research and Development, journalPhotoCn=k2l8zgqwstuxxevFX2qxVA==, journalPhotoEn=r6k0Ry+RD+c/VXj5QarmDw==, journalFirstLetter=J, journalRecommend=null, journalNew=null, journalCollection=null, jcrJf=null, cjcrJf=null, jcrJfStr=null, cjcrJfStr=null, submissionFirstDecision=null, sciSubjectClassification=null, casSubjectClassification=null, citeScore=null, totalCitationFrequency=null, icpCode=null, psCode=null, advertisingLicenseCode=null, copyrightInformation=null, country=null, option=, provinceCode=null, provinceName=null, collectFlag=false, interPubPlatform=, interPubPlatformUrl=null), detailUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202330966, detailUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202330966, pdfUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/PDF/10.7544/issn1000-1239.202330966, pdfUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/PDF/10.7544/issn1000-1239.202330966, aliStartDate=0, aliEndDate=0, collectionFlag=false, citedCount=null, citedUrl=null, previewStatus=0, delFlag=0, hasFullText=1, orderTime=1714492800000, fullTextJson=null, articleText=null, reference=null)
收藏切换
一种基于安全多方计算的快速Transformer安全推理方案
刘伟欣 1 , 管晔玮 1 , 霍嘉荣 1 , 丁元朝 1 , 郭华 1, 2, * , 李博 2
计算机研究与发展 | 大模型与安全专题 2024,61(5): 1218-1229
收起
收藏切换
计算机研究与发展 |大模型与安全专题 2024 , 61 (5) : 1218 -1229
一种基于安全多方计算的快速Transformer安全推理方案
全屏
[Author(id=1329461185600139583, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=sy2339130lwx@buaa.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329461185663054145, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, authorId=1329461185600139583, language=EN, stringName=Weixin Liu, firstName=Weixin, middleName=null, lastName=Liu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1School of Cyber Science and Technology, Beihang University, Beijing 100191, bio={"content":"

born in 2001. Master candidate. His main research interests include privacy-preserving machine learning, cryptography

"}, bioImg=null, bioContent=

born in 2001. Master candidate. His main research interests include privacy-preserving machine learning, cryptography

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329461185734357314, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, authorId=1329461185600139583, language=CN, stringName=刘伟欣, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1北京航空航天大学网络空间安全学院 北京 100191, bio={"img":"w14iN/nyXmi0b5gI7jd+Tg==","content":"

2001年生. 硕士研究生. 主要研究方向为隐私保护机器学习、密码学

"}, bioImg=w14iN/nyXmi0b5gI7jd+Tg==, bioContent=

2001年生. 硕士研究生. 主要研究方向为隐私保护机器学习、密码学

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329461185436561720, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, xref=1, ext=[AuthorCompanyExt(id=1329461185449144633, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, companyId=1329461185436561720, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1School of Cyber Science and Technology, Beihang University, Beijing 100191), AuthorCompanyExt(id=1329461185457533242, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, companyId=1329461185436561720, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1北京航空航天大学网络空间安全学院 北京 100191)])]), Author(id=1329461185814049092, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329461185876963654, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, authorId=1329461185814049092, language=EN, stringName=Yewei Guan, firstName=Yewei, middleName=null, lastName=Guan, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1School of Cyber Science and Technology, Beihang University, Beijing 100191, bio={"content":"

born in 1999. PhD candidate. His main research interests include secure multi-party computation and privacy-preserving machine learning

"}, bioImg=null, bioContent=

born in 1999. PhD candidate. His main research interests include secure multi-party computation and privacy-preserving machine learning

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329461185948266823, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, authorId=1329461185814049092, language=CN, stringName=管晔玮, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1北京航空航天大学网络空间安全学院 北京 100191, bio={"img":"n8eryvL6alcGtG8ngVuApA==","content":"

1999年生. 博士研究生. 主要研究方向为安全多方计算、隐私保护机器学习

"}, bioImg=n8eryvL6alcGtG8ngVuApA==, bioContent=

1999年生. 博士研究生. 主要研究方向为安全多方计算、隐私保护机器学习

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329461185436561720, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, xref=1, ext=[AuthorCompanyExt(id=1329461185449144633, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, companyId=1329461185436561720, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1School of Cyber Science and Technology, Beihang University, Beijing 100191), AuthorCompanyExt(id=1329461185457533242, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, companyId=1329461185436561720, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1北京航空航天大学网络空间安全学院 北京 100191)])]), Author(id=1329461186019569993, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329461186090873163, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, authorId=1329461186019569993, language=EN, stringName=Jiarong Huo, firstName=Jiarong, middleName=null, lastName=Huo, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1School of Cyber Science and Technology, Beihang University, Beijing 100191, bio={"content":"

born in 2002. Undergraduate. His main research interest includes applied cryptography

"}, bioImg=null, bioContent=

born in 2002. Undergraduate. His main research interest includes applied cryptography

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329461186153787724, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, authorId=1329461186019569993, language=CN, stringName=霍嘉荣, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1北京航空航天大学网络空间安全学院 北京 100191, bio={"img":"OK3mx3xIMhIvXPEwJOiMsA==","content":"

2002年生. 本科生. 主要研究方向为应用密码学

"}, bioImg=OK3mx3xIMhIvXPEwJOiMsA==, bioContent=

2002年生. 本科生. 主要研究方向为应用密码学

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329461185436561720, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, xref=1, ext=[AuthorCompanyExt(id=1329461185449144633, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, companyId=1329461185436561720, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1School of Cyber Science and Technology, Beihang University, Beijing 100191), AuthorCompanyExt(id=1329461185457533242, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, companyId=1329461185436561720, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1北京航空航天大学网络空间安全学院 北京 100191)])]), Author(id=1329461186233479502, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, orderNo=3, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329461186313171280, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, authorId=1329461186233479502, language=EN, stringName=Yuanchao Ding, firstName=Yuanchao, middleName=null, lastName=Ding, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1School of Cyber Science and Technology, Beihang University, Beijing 100191, bio={"content":"

born in 1999. Master candidate. His main research interests include privacy-preserving machine learning and cryptography

"}, bioImg=null, bioContent=

born in 1999. Master candidate. His main research interests include privacy-preserving machine learning and cryptography

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329461186380280145, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, authorId=1329461186233479502, language=CN, stringName=丁元朝, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, address=1北京航空航天大学网络空间安全学院 北京 100191, bio={"img":"LYSy2heXEILnbgGIz9MmrQ==","content":"

1999年生. 硕士研究生. 主要研究方向为隐私保护机器学习、密码学

"}, bioImg=LYSy2heXEILnbgGIz9MmrQ==, bioContent=

1999年生. 硕士研究生. 主要研究方向为隐私保护机器学习、密码学

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329461185436561720, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, xref=1, ext=[AuthorCompanyExt(id=1329461185449144633, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, companyId=1329461185436561720, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1School of Cyber Science and Technology, Beihang University, Beijing 100191), AuthorCompanyExt(id=1329461185457533242, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, companyId=1329461185436561720, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1北京航空航天大学网络空间安全学院 北京 100191)])]), Author(id=1329461186472554835, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, orderNo=4, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=hguo@buaa.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=1, authorType=1, ext={EN=AuthorExt(id=1329461186548052310, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, authorId=1329461186472554835, language=EN, stringName=Hua Guo, firstName=Hua, middleName=null, lastName=Guo, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, *, address=1School of Cyber Science and Technology, Beihang University, Beijing 100191
2State Key Laboratory of Complex & Critical Software Environment (Beihang University), Beijing 100878, bio={"content":"

born in 1980. PhD, associate professor. Member of CCF. Her main research interests include privacy-preserving machine learning and cryptography

"}, bioImg=null, bioContent=

born in 1980. PhD, associate professor. Member of CCF. Her main research interests include privacy-preserving machine learning and cryptography

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329461186623549783, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, authorId=1329461186472554835, language=CN, stringName=郭华, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=1, 2, *, address=1北京航空航天大学网络空间安全学院 北京 100191
2复杂关键软件环境全国重点实验室(北京航空航天大学) 北京 100878, bio={"img":"98sFdXx5RqbSU2tgXeaOdQ==","content":"

1980年生. 博士,副教授. CCF会员. 主要研究方向为隐私保护机器学习、密码学

"}, bioImg=98sFdXx5RqbSU2tgXeaOdQ==, bioContent=

1980年生. 博士,副教授. CCF会员. 主要研究方向为隐私保护机器学习、密码学

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329461185436561720, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, xref=1, ext=[AuthorCompanyExt(id=1329461185449144633, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, companyId=1329461185436561720, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1School of Cyber Science and Technology, Beihang University, Beijing 100191), AuthorCompanyExt(id=1329461185457533242, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, companyId=1329461185436561720, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=1北京航空航天大学网络空间安全学院 北京 100191)]), AuthorCompany(id=1329461185524642107, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, xref=2, ext=[AuthorCompanyExt(id=1329461185533030716, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, companyId=1329461185524642107, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2State Key Laboratory of Complex & Critical Software Environment (Beihang University), Beijing 100878), AuthorCompanyExt(id=1329461185541419325, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, companyId=1329461185524642107, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2复杂关键软件环境全国重点实验室(北京航空航天大学) 北京 100878)])]), Author(id=1329461188242551129, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, orderNo=5, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1329461188372574555, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, authorId=1329461188242551129, language=EN, stringName=Bo Li, firstName=Bo, middleName=null, lastName=Li, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2State Key Laboratory of Complex & Critical Software Environment (Beihang University), Beijing 100878, bio={"content":"

born in 1981. PhD, associate professor. Member of CCF. His main research interests include privacy-preserving machine learning and network security

"}, bioImg=null, bioContent=

born in 1981. PhD, associate professor. Member of CCF. His main research interests include privacy-preserving machine learning and network security

, aboutCorrespAuthor=null), CN=AuthorExt(id=1329461188427100508, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, authorId=1329461188242551129, language=CN, stringName=李博, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=2, address=2复杂关键软件环境全国重点实验室(北京航空航天大学) 北京 100878, bio={"img":"r6ckzWpanNAIEVm/Ev6JLQ==","content":"

1981年生. 博士,副教授. CCF会员. 主要研究方向为隐私保护机器学习、网络安全

"}, bioImg=r6ckzWpanNAIEVm/Ev6JLQ==, bioContent=

1981年生. 博士,副教授. CCF会员. 主要研究方向为隐私保护机器学习、网络安全

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1329461185524642107, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, xref=2, ext=[AuthorCompanyExt(id=1329461185533030716, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, companyId=1329461185524642107, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2State Key Laboratory of Complex & Critical Software Environment (Beihang University), Beijing 100878), AuthorCompanyExt(id=1329461185541419325, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1329461181250646298, companyId=1329461185524642107, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=2复杂关键软件环境全国重点实验室(北京航空航天大学) 北京 100878)])])]
刘伟欣1 , 管晔玮1, 霍嘉荣1, 丁元朝1, 郭华1, 2, * , 李博2
作者信息
  • 1北京航空航天大学网络空间安全学院 北京 100191
  • 2复杂关键软件环境全国重点实验室(北京航空航天大学) 北京 100878
通讯作者:
郭华()
作者简介:

2001年生. 硕士研究生. 主要研究方向为隐私保护机器学习、密码学

1999年生. 博士研究生. 主要研究方向为安全多方计算、隐私保护机器学习

2002年生. 本科生. 主要研究方向为应用密码学

1999年生. 硕士研究生. 主要研究方向为隐私保护机器学习、密码学

1980年生. 博士,副教授. CCF会员. 主要研究方向为隐私保护机器学习、密码学

1981年生. 博士,副教授. CCF会员. 主要研究方向为隐私保护机器学习、网络安全

A Fast and Secure Transformer Inference Scheme with Secure Multi-Party Computation
Weixin Liu1 , Yewei Guan1, Jiarong Huo1, Yuanchao Ding1, Hua Guo1, 2, * , Bo Li2
Affiliations
  • 1School of Cyber Science and Technology, Beihang University, Beijing 100191
  • 2State Key Laboratory of Complex & Critical Software Environment (Beihang University), Beijing 100878
  • born in 2001. Master candidate. His main research interests include privacy-preserving machine learning, cryptography

    born in 1999. PhD candidate. His main research interests include secure multi-party computation and privacy-preserving machine learning

    born in 2002. Undergraduate. His main research interest includes applied cryptography

    born in 1999. Master candidate. His main research interests include privacy-preserving machine learning and cryptography

    born in 1980. PhD, associate professor. Member of CCF. Her main research interests include privacy-preserving machine learning and cryptography

    born in 1981. PhD, associate professor. Member of CCF. His main research interests include privacy-preserving machine learning and network security

出版时间: 2024-05-01 doi: 10.7544/issn1000-1239.202330966
文章导航
收藏切换

Transformer模型在自然语言处理、计算机视觉等众多领域得到了广泛应用,并且有着突出的表现. 在Transformer的推理应用中用户的数据会被泄露给模型提供方. 随着数据隐私问题愈发得到公众的关注,上述数据泄露问题引发了学者们对Transformer安全推理的研究,使用安全多方计算(secure multi-party computation,MPC)实现Transformer模型的安全推理是当前的一个研究热点. 由于Transformer模型中存在大量非线性函数,因此使用MPC技术实现Transformer安全推理会造成巨大的计算和通信开销. 针对Transformer安全推理过程中开销较大的Softmax注意力机制,提出了2种MPC友好的注意力机制Softmax freeDiv Attention和2Quad freeDiv Attention. 通过将Transformer模型中的Softmax注意力机制替换为新的MPC友好的注意力机制,同时结合激活函数GeLU的替换以及知识蒸馏技术,提出了一个MPC友好的Transformer转换框架,通过将Transformer模型转化为MPC友好的Transformer模型,提高Transformer安全推理的效率. 在局域网环境下使用安全处理器(secure processing unit, SPU)提供的隐私计算协议,基于所提出的MPC友好的Transformer转换框架,在SST-2上使用Bert-Base进行安全推理. 测试结果表明,在保持推理准确率与无近似模型一致的情况下,安全推理计算效率提高2.26倍.

安全推理  /  Transformer  /  安全多方计算  /  安全处理器  /  知识蒸馏

Transformer has been widely used in many fields such as natural language processing and computer vision, and has outstanding performance. The users’ data will be leaked to the Transformer model provider during inference. With the increasing public attention on data privacy, the above data leakage problem has triggered researchers’ study on secure Transformer inference. Implementing secure Transformer inference with secure multi-party computation (MPC) is today’s hot topic. Due to the widely existence of non-linear functions in Transformer, it is hard to use MPC to implement secure Transformer inference, which leads to huge computation and communication cost. We focus on Softmax attention, bottleneck in secure Transformer inference, and propose two kinds of MPC-friendly attention mechanism, Softmax freeDiv Attention and 2Quad freeDiv Attention. By replacing the Softmax attention in Transformer with the MPC-friendly attention mechanism proposed, combining with the replacement of activation function GeLU and knowledge distillation, we propose an MPC-friendly Transformer convert framework, which can convert Transformer model to an MPC-friendly one, so as to improve the performance of secure Transformer inference later. Based on the proposed MPC-friendly Transformer convert framework , we perform secure Bert-Base inference on SST-2 in the LAN setting, using privacy computing protocols provided by secure processing unit (SPU). The result shows that the secure inference achieves 2.26 times speedup while maintaining the accuracy with non-approximation model.

secure inference  /  Transformer  /  secure multi-party computation (MPC)  /  secure processing unit (SPU)  /  knowledge distillation
刘伟欣, 管晔玮, 霍嘉荣, 丁元朝, 郭华, 李博. 一种基于安全多方计算的快速Transformer安全推理方案. 计算机研究与发展, 2024 , 61 (5) : 1218 -1229 . DOI: 10.7544/issn1000-1239.202330966
Weixin Liu, Yewei Guan, Jiarong Huo, Yuanchao Ding, Hua Guo, Bo Li. A Fast and Secure Transformer Inference Scheme with Secure Multi-Party Computation[J]. Journal of Computer Research and Development, 2024 , 61 (5) : 1218 -1229 . DOI: 10.7544/issn1000-1239.202330966
近年来机器学习领域的Transformer模型飞速发展,其凭借巨大的参数规模和特殊的网络结构在众多领域有着出色的表现,包括自然语言处理[1-2](natural language processing, NLP)、计算机视觉[3-4](computer vision, CV)等. 在 Transformer的应用中,通常有3类参与方,第1类是数据提供方,其提供在模型训练过程中使用到的训练样本;第2类是模型提供方,其使用训练样本进行模型训练从而得到一个训练好的模型,之后可以提供推理服务;第3类是模型使用方,其申请使用模型提供方所提供的推理服务,输入数据并得到推理结果. 以OpenAI推出的ChatGPT为例,其数据来源于互联网上几乎所有的公开文本数据,因此数据提供方可看作是互联网上的所有用户,再由OpenAI作为模型提供方在庞大的训练数据下进行模型训练,得到ChatGPT模型,最后将其公开以供各用户使用,此时使用ChatGPT的用户身份即为模型使用方,如图1所示.
近年来数据隐私安全问题逐渐得到社会关注,Transformer模型的隐私保护问题成为一个研究热点[5-9]. 一方面,在模型训练过程中,模型提供方会收集大规模训练样本进行模型训练,但实际上这些训练样本有可能来自于互联网上的大量用户群体,或是某企业对应的用户群体,在这个过程中用户有权保护自己个人数据的隐私性. 另一方面,在模型推理过程中,用户更倾向于保护自己的输入数据及推理结果的隐私性,尤其是医疗诊断、图像识别等领域,用户的输入数据通常会涉及个人的隐私信息,同时模型提供方也需要保护自己的模型参数不会泄露,因为该模型是收集了大量的训练数据并消耗了大量的算力而训练得到的.
隐私保护机器学习(privacy preserving machine learning, PPML)使用同态加密(homomorphic encryption, HE)[10-11]、安全多方计算(secure multi- party computation, MPC)[9,12-13]、差分隐私(differential privacy, DP)[14]、可信执行环境(trusted execution environment, TEE)[15]等技术对机器学习模型进行隐私保护,可以保证在训练过程中数据提供方的原始数据不会泄露给模型提供方,在推理过程中模型提供方的模型参数和用户的输入输出数据不会被泄露. 对于决策树[16]、支持向量机[17]、神经网络[12-13,18-19]等机器学习算法的隐私保护技术已有大量研究.
近年来随着Transformer的提出,以ChatGPT为代表的Transformer模型愈发得到人们的关注. 在Transformer推理中,模型提供方希望模型参数不会泄露,模型使用方希望其模型推理的输入和输出不会泄露. 因此,Transformer安全推理问题引发了众多学者的研究[6-9,20-22]. MPC是Transformer安全推理中常用的密码技术,但其带来的额外开销往往是难以接受的,例如,输入长度为512的序列,一个12层的BERT-Base模型在MPC系统中进行安全推理需要59.0 s,而在明文下只需要不到1 s[21]. 因此,如何在保证准确率不受影响的前提下减少Transformer安全推理所需的时间成为了研究热点.
机器学习模型可划分为线性函数和非线性函数,对于复杂的非线性函数的处理方法一般是使用一个新的函数$f' $来替换原复杂的非线性函数. 对于$f' $的选择通常有2种思路:一种是近似方法,即通过函数拟合的方法得到一个分段多项式函数,将其作为非线性函数的替换,代表工作有ABY3[18],Delphi[23]等,该方法能够保持较高的准确率;另一种是启发式方法,使用具有与原函数相似性质的MPC友好函数进行替换,例如使用Quad替换GeLU,代表工作有MPCFormer[21],SecureML[12]等,这类方法相比于前一种方法在计算效率上通常会更加高效,但需要额外考虑准确率下降的问题,因此需要在后续添加微调步骤,以保证模型的可用性.
目前对于Transformer安全推理工作的一大研究方向是将Transformer安全推理中开销较大的函数替换为MPC友好的函数,再通过知识蒸馏等技术提升模型准确率,最后在MPC友好的Transformer模型上实现安全推理. 以Transformer模型的一种变体BERT-Base为例,在BERT-Base模型的基于MPC技术的安全推理中,Softmax和GeLU占据了大量运行时间[21]. 例如文献[21]中指出对于一个12层的BERT-Base模型,在输入序列长度为512时,Softmax和GeLU耗时分别占总时长的67.8%和18.6%,二者占据了安全推理过程中86.4%的时长,因此在Transformer安全推理中对Softmax注意力机制和激活函数GeLU的效率进行优化,可以有效减少Transformer安全推理的总执行耗时. 目前对于MPC友好的函数替换方法的主要研究为如何在权衡效率和准确率的同时选择合适的函数替换方案.
文献[21]对Transformer模型进行了函数替换,针对Softmax注意力机制首先选择2ReLU Attention[12]的替换方法并进行测试,在其基础上又提出了2Quad Attention的替换方法;针对激活函数GeLU,文献[21]提出了平方函数Quad的替换方法. 文献[22]对Vision Transformer模型进行了函数替换,针对Softmax注意力机制选择并测试了目前已有的一系列注意力机制替换方法,包括:2ReLU Attention[12],2Quad Attention[21],Scaling Attention[24],Linformer Attention[25]等,在其中选择了2ReLU Attention,Scaling Attention这2种作为其替换函数.
然而,文献[21-22]中提出的函数替换方法并没有达到效率和准确率的平衡. Scaling Attention等替换方法效率较高,但准确率下降较大;2ReLU Attention和2Quad Attention准确率较高,但同时也导致了较大的开销. 本文针对Softmax注意力机制给出一种新的函数替换方法,相比2ReLU Attention,2Quad Attention可以省去非MPC友好的倒数运算,同时能保持较高的准确率,达到了效率与准确率的平衡.
本文的主要贡献包括2个方面:
1)提出了2种新的MPC友好的注意力机制Softmax freeDiv Attention和2Quad freeDiv Attention,在几乎不影响准确率的前提下减少了Transformer中的注意力机制在MPC场景中的计算量,进而提高了基于MPC技术的Transformer安全推理速度;
2)将提出的MPC友好的注意力机制与知识蒸馏技术结合,提出了一个MPC友好的Transformer转换框架,用于将Transformer模型转换为MPC友好的Transformer模型. 对转换得到的Transformer模型在隐语安全处理器(secure processing unit, SPU)下进行安全推理,在SST-2任务上可以在保证准确率相比于无近似模型不受影响的前提下将安全推理速度提升2.26倍.
目前Transformer安全推理工作的主要研究目标是在保证准确率的前提下提高计算效率. 在函数近似方法方面,2022年Hao等人[6]提出的Iron基于同态加密和秘密分享技术实现各函数的隐私保护协议,从而构建了Transformer安全推理框架,对于其中的矩阵乘法,在Cheetah[13]中基于同态加密方法的基础上进行优化,达到了目前最优的开销. Iron将非线性函数拆解为若干基础原语,并调用Sirnn[26]实现. 2023年Zheng等人[8]提出的Primer直接将Transformer的非线性算子使用混淆电路实现. 同年Puma[27]和Privformer[28]分别基于ABY3[18]和Falcon[29]框架实现了Transformer的安全推理,其主要工作是针对Transformer中独有的算子利用对应框架提供的协议设计对应的隐私保护协议. 2023年文献[20]对GeLU进行分段拟合,使用Sirnn中的多项技术来优化近似多项式的安全计算. 同年,文献[9]基于函数秘密分享技术构造了GeLU,Softmax等非线性函数的安全计算协议,大大提升了安全推理的效率.
在启发式方法方面,2022年Chen等人[7]提出的THE-X对于非线性部分使用ReLU函数替换GeLU,使用一个3层全连接网络近似Softmax,但其中ReLU的计算则由数据持有者在明文上计算,从而导致中间结果泄露给数据持有者,进而可能导致模型持有者的模型参数相关信息遭到泄露. 同时 THE-X使用知识蒸馏技术对模型进行了微调,提升了模型准确率. 同年,Li等人[21]提出MPCFormer,该模型的主要思想是首先将Transformer中开销较大的函数即Softmax和GeLU替换为MPC友好的函数,之后使用知识蒸馏技术对该模型进行微调,提升模型准确率,最后使用现有MPC技术实现更新后的模型的安全推理. MPCViT[22]的思路与MPCFormer略有不同,其指出在Vision Transformer(ViT)中并不是所有的注意力机制都同样重要,于是首先分析了多种注意力机制的效率与准确率(效率较高的注意力机制通常具有较低的准确率),将所有的Softmax注意力机制替换为一种准确率相对较高但效率相对较低的MPC友好的注意力机制2ReLU Attention;之后使用神经架构搜索(neural architecture search, NAS)技术将部分2ReLU Attention注意力机制替换为准确率相对较低但效率相对较高的注意力机制Scaling Attention,保证在不过多降低准确率的前提下尽可能达到最高的计算效率,最后再使用知识蒸馏技术提升模型准确率.
在现有的基于MPC的PPML中,有许多研究通过MPC友好的函数替换来加快安全推理速度. 早在2017年就有SecureML方案[12]将Softmax中的指数函数替换为ReLU,因为在其针对的机器学习模型中Softmax仅在最后一层被使用,因此其对准确率几乎不造成影响. 2018年Chou等人[30]提出的Faster Cryptonets使用二次函数替换了神经网络中的ReLU,但其会导致梯度下降偏离,从而导致模型准确率大幅度下降. 2020年的Delphi方案[23]通过NAS技术将神经网络中的部分ReLU替换为平方函数.
在Transformer安全推理的工作中,THE-X方案[7] 对于Softmax使用一个3层全连接网络来进行替换. MPCFormer[21]将Transformer中的Softmax注意力机制和激活函数GeLU分别替换为MPC友好的注意力机制和激活函数. 通过实验,MPCFormer得出在GLUE数据集下分别将注意力机制和激活函数GeLU替换为MPC友好的2Quad Attention注意力机制和平方函数Quad,在准确率下降幅度不大的前提下获得2.2倍的安全推理速度. MPCViT[22]针对ViT,指出THE-X[7]和MPCFormer[21]中的注意力机制替换方法并不灵活,直接应用在ViT中会导致准确率大幅度下降. 因此,针对不同特点的注意力机制替换方法,MPCViT使用NAS技术选择对Transformer中注意力机制的不同头级(head-wise)使用不同的替换,而不是像MPCFormer中对所有的注意力机制使用相同的替换.
在本节中,首先介绍Transformer和知识蒸馏的概念,然后介绍Transformer安全推理框架.
Transformer采用Encoder-Decoder结构,由于本文主要针对BERT这类仅包含Encoder的Transformer模型(Encoder-only Transformer)进行研究,本节以BERT模型为例,介绍Encoder-only Transformer结构,如图2所示.
Transformer的输入首先会经过一个Embedding层,输出词向量X,之后的Encoder部分包含多个Transformer块,在每个Transformer块中包含3部分:注意力机制(Attention)、前馈神经网络(feed forward network, FFN)、2个层归一化(LN). 这3部分详细介绍如下:
1)注意力机制(Attention).注意力机制有助于模型关注输入序列中的不同部分,输入矩阵Q=XWQ, K=XWK, V=XWV,其中WQ, WK, WV表示权值矩阵,令d表示向量维度,则注意力机制如式(1)所示.
$ {{Attention}}({\boldsymbol{Q}},{\boldsymbol{K}},{\boldsymbol{V}}) = {{softmax}}\left( {\frac{{{\boldsymbol{Q}}{{\boldsymbol{K}}^{\text{T}}}}}{{\sqrt {{d}} }}} \right){\boldsymbol{V}} , $
其中softmax如式(2)所示.
$ \begin{gathered}{{softmax}}\left( {{x_0},{x_1}, … ,{x_{n - 1}}} \right) = \\\frac{1}{{\displaystyle\sum\limits_{i = 0}^{n - 1} {{{\text{e}}^{{x_i}}}} }}\left( {{{\text{e}}^{{x_0}}},{{\text{e}}^{{x_1}}}, … ,{{\text{e}}^{{x_{n - 1}}}}} \right) \\ \end{gathered} .$
多头注意力机制(multi-head attention, MHA)将上述注意力机制扩展到H个并行的注意力层,如式(3)所示.
$ \begin{split}&{{MultiHead}}({\boldsymbol{Q}},{\boldsymbol{K}},{\boldsymbol{V}}) = \\&{{Concat}}\left( {{{Attention}}\left( {{{\boldsymbol{Q}}_i},{{\boldsymbol{K}}_i},{{\boldsymbol{V}}_i}} \right)} \right){{\boldsymbol{W}}^O},i = 1,2,… ,H.\end{split} $
2)前馈神经网络(FFN). FFN包含2个线性层和中间的一个激活函数,如式(4)所示.
$ {{FFN}}({\boldsymbol{X}}) = {{GeLU}}({\boldsymbol{X}}{{\boldsymbol{W}}_1} + {{\boldsymbol{B}}_1}){{\boldsymbol{W}}_2} + {{\boldsymbol{B}}_2}, $
其中GeLU近似如式(5)所示.
$ {{GeLU}}(x) = 0.5x\left[ {1 + \tanh \left( {\sqrt {\frac{2}{{\text{π }}}} \left( {x + 0.047715{x^3}} \right)} \right)} \right].$
3)层归一化(LN). LN需计算输入的均值和方差,对于输入${\boldsymbol{x}}=(x_0,\;x_1,\;…,x_{n-1})$,LN如式(6)所示.
$ \begin{aligned}&y_j=\gamma_j\cdot \frac{x_j-\mu }{\sqrt{\sigma^2+\varepsilon} }+\beta _j, \\ &\mu =\frac{1}{n}\sum_{j=0}^{n-1}x_j,\;\sigma^2=\frac{1}{n} \sum\limits_{j=0}^{n-1}(x_j-\mu )^2.\end{aligned} $
知识蒸馏[31]是模型压缩技术中的一种,用于解决小的模型训练困难的问题. 知识蒸馏技术允许一个小的学生模型S学习一个大的教师模型T的知识. 具体来说,学生模型在训练过程中会去模仿教师模型的行为. 教师模型的行为能提供比训练数据标签更多的信息,学生模型通过学习这些信息能够获得教师模型的泛化能力. 模型的行为可以用行为函数描述,在Transformer模型的蒸馏中,MHA层、FFN层或者一些中间表示层都可以看作行为函数. 用f S,f T分别表示学生模型和教师模型的行为函数,则知识蒸馏的过程可以描述为最小化目标函数式(7).
$ {{L}}_{\rm KD}={{{\displaystyle\sum\limits_{{\boldsymbol{d}}_i\in D}}}L({f}^{{\mathrm{S}}}({\boldsymbol{d}}_i),{f}^{{\mathrm{T}}}({\boldsymbol{d}}_i))} , $
其中L(·)是评估教师模型与学生模型之间差异的损失函数,D表示训练数据集,di是一条文本输入.
本文沿用文献[21]所给出的Transformer安全推理框架,整体架构如图3所示,主要分为3个部分:函数替换、知识蒸馏、安全推理,其中前2部分共同组成了MPC友好的Transformer转换框架.
1)函数替换. 输入预训练的Transformer模型,将其中开销较大的函数替换为MPC友好的函数,该步骤可提升后续安全推理的推理速度.
2)知识蒸馏. 以无近似模型为教师模型,替换后的模型为学生模型,使用知识蒸馏技术对替换后的模型进行微调,得到更新参数后的MPC友好的近似模型,该步骤可以提升后续安全推理的准确率.
3)安全推理. 服务端以更新参数后的MPC友好的近似模型作为输入,客户端以其隐私数据作为输入,双方在不泄露各自输入的安全保证下执行MPC协议进行安全推理.
原始Transformer预训练模型在准确率上表现良好,但由于其中的Softmax注意力机制和激活函数GeLU在使用MPC技术实现时效率较低,导致模型推理速度较慢. 本节提出一种MPC友好的Transformer转换框架,该框架可将预训练的Transformer模型转换为MPC友好的Transformer模型,同时保证模型准确率不受影响,并且提高安全推理的推理速度,从而可用于Transformer安全推理.
MPC友好的Transformer转换框架分为函数替换和知识蒸馏2个阶段,工作流程如图4所示. 第1阶段为函数替换,将Softmax注意力机制替换为本文提出的MPC友好的Softmax freeDiv Attention或2Quad freeDiv Attention(两者统称为freeDiv Attention),同时将前馈神经网络中的激活函数GeLU替换为MPC友好的ReLU函数或平方函数Quad,可提高推理速度. 由于该替换改变了注意力机制和激活函数的功能,直接使用原始模型的参数会导致准确率下降. 第2阶段的知识蒸馏技术以原模型为教师模型,以上一步替换后的模型为学生模型,对替换后的模型使用知识蒸馏技术进行微调,更新模型参数使其与新的注意力机制和激活函数适配,从而提高了模型的准确率.
通过上述2个阶段,本文提出的MPC友好的Transformer转换框架实现了预训练Transformer模型到MPC友好的Transformer模型的转换.
在函数替换阶段,首先分析在Transformer安全推理过程中的主要开销,通过将开销较大的函数替换为MPC友好的函数可以提高安全推理的效率. MPCFormer[21]和MPCViT[22]分别通过实验指出Softmax和GeLU占据了Transformer推理的大部分时长. 因此本文主要考虑分别将Softmax注意力机制和激活函数GeLU替换为MPC友好的注意力机制和激活函数. 对于Softmax注意力机制,其中包含3种非线性运算:指数运算、比较运算和倒数运算,均需通过MPC协议安全计算. 文献[21]中给出的2Quad Attention替换方案去除了其中的指数运算和比较运算,并且在SST-2任务上达到较好的准确率,但其中还存在较大开销的倒数运算. 本文考虑使用一个可以在明文下计算的函数得到倒数运算中的输入,从而去除安全倒数运算,进一步减少开销. 对于激活函数GeLU,文献[21]中已经给出性能和准确率都表现较优的替换函数,本文将直接使用这些函数进行替换. 具体的替换方案将在第4节介绍.
在知识蒸馏阶段中,本文使用无近似模型M作为教师模型,近似后的模型$M' $作为学生模型,在下游任务数据集上进行知识蒸馏. 初始化时学生模型$M' $使用与教师模型M相同的模型参数,二者仅在使用的函数上不同.
蒸馏任务执行时分为2个步骤:在第1步中学生模型主要学习教师模型在Transformer块的行为;在第2步中学生模型主要学习教师模型在最后的预测层的行为. 下面具体描述2个步骤:
第1步,学生模型将学习教师模型在每个Transformer块的注意力矩阵以及每一个Transformer块之后的隐藏状态信息,这些信息中包含了大量语言知识. 蒸馏过程中向教师模型和学生模型输入同样一组训练数据,并分别计算2个模型在每一层的注意力矩阵以及每一层之后的隐藏状态信息. 使用均方误差作为损失函数,计算如式(8)所示.
$ {{{L}}_1} = \sum\limits_{{{{\boldsymbol{d}}_i}} \in {{D}}} {(MSE(} {\boldsymbol{at}}{{\boldsymbol{t}}^{\rm{S}}},{\boldsymbol{at}}{{\boldsymbol{t}}^{\rm{T}}}) + MSE({\boldsymbol{re}}{{\boldsymbol{p}}^{\rm{S}}},{\boldsymbol{re}}{{\boldsymbol{p}}^{\rm{T}}})),$
$ \begin{gathered}MSE({\boldsymbol{Y}},{\boldsymbol{\hat Y}}) = \frac{1}{n}\sum\limits_{i = 1}^n {{{({{\boldsymbol{Y}}_i} - \hat {{{\boldsymbol{Y}}_i}})}^2}} ,\end{gathered} $
其中${\boldsymbol{att}}^{\mathrm{S}} $,${\boldsymbol{att}}^{\mathrm{T}} $分别表示学生模型和教师模型的注意力矩阵,${\boldsymbol{rep}}^{\mathrm{S}} $,${\boldsymbol{rep}}^{\mathrm{T}} $分别表示学生模型和教师模型在Transformer块之后的隐藏状态信息.
第2步,学生模型将学习教师模型在最后的预测层输出. 同样在蒸馏过程中向教师模型和学生模型输入同样一组训练数据并分别计算这2个模型在预测层的输出. 损失函数设置为学生模型logits输出和教师模型logits输出之间的软交叉熵损失,如式(9)所示.
$ {{L}_2} = CE({\boldsymbol{logi}}{{\boldsymbol{t}}^{\mathrm{T}}}/t,{\boldsymbol{logi}}{{\boldsymbol{t}}^{\mathrm{S}}}/t) , $
其中${\boldsymbol{logit}}^{\mathrm{T}} $和${\boldsymbol{logit}}^{\mathrm{S}} $分别表示学生模型和教师模型的logits向量,CE表示交叉熵损失,t是温度. 在本文的实验过程中,设置t = 1时有较好的蒸馏效果.
本文针对Transformer模型中的Softmax注意力机制和激活函数GeLU给出MPC友好的函数替换方案. 对Softmax注意力机制替换时,本文提出了2种新的注意力机制Softmax freeDiv Attention和2Quad freeDiv Attention,其中2Quad freeDiv Attention在计算效率上可持平目前计算效率最高的注意力机制Scaling Attention,并且不会导致Scaling Attention中存在的准确率大幅度下降问题. 对激活函数GeLU替换时,本文使用文献[21]中提到的2种替换方案.
对于注意力机制的替换,以往采取的方法包括:将Softmax表达式中的指数函数替换为ReLU的2ReLU Attention、将指数函数替换为平方函数的2Quad Attention、将整个注意力机制替换为矩阵乘法的Scaling Attention等,其中$ReLU(\hat {x}) =\max(\hat{x},0)$.
上述替换在准确率和效率上各有优劣,如2ReLU Attention相比于原始的Softmax Attention不会导致准确率大幅度下降,但会使计算效率降低. Scaling Attention的表达式为
$ {{ScaleAttn}}({\boldsymbol{Q}},{\boldsymbol{K}},{\boldsymbol{V}}) = \frac{1}{n}\left( {{\boldsymbol{Q}}{{\boldsymbol{K}}^{\text{T}}}} \right){\boldsymbol{V}} , $
其中n为输入序列长度. 与式(1)相比,式(10)中省去Softmax的计算过程,因此计算高效,但会导致推理阶段准确率大幅度降低.
MPCViT[22]中总结了Softmax注意力机制的3个重要特性,即单调性、非负性、求和为1. 上述2ReLU Attention、2Quad Attention的主要思想是将注意力机制中的指数函数替换为计算较为简单的其他函数. 因为在基于MPC技术的Transformer安全推理中通常要将计算指数函数近似为分段多项式,再使用不经意分段多项式求值,需调用多次安全比较协议和安全乘法协议,而将指数函数替换为ReLU或平方函数分别只需要1次比较+1次乘法和1次乘法,大大提高了安全推理的效率. 其中的ReLU和平方函数本身均能保证单调性和非负性(其中单调性只需要在数据分布区间上满足),最后再通过将每一项除以所有项求和的方式来满足求和为1.
目前的注意力机制替换方法将指数函数替换为MPC友好的ReLU或平方函数,能够在满足Softmax注意力机制的前2个性质的前提下大大减少运算量,从而提高计算效率. 但为了满足第3个性质,需将每一项除以所有项的求和. 对于一个规模为n的张量,使用MPC技术时需执行n次秘密值之间的除法. 由于在MPC中除法的计算开销远大于加法,因此通常先计算所有项求和的倒数,再计算n次乘法,共需1次除法与n次乘法的开销. 若使用一个常数来近似分母(即各分项的求和),将注意力机制的第3个特性“求和为1”放宽到“求和近似于1”,那么可以在保证准确率的前提下去除倒数运算,减少开销,然而难点在于如何选择这个近似的常数.
在选择注意力机制中分母近似常数时,本文首先使用预训练的Transformer模型对数据集进行多次推理,将每次推理过程中不同输入序列长度下的分母均值进行记录,利用这些数据可以拟合得到一个关于输入序列长度的函数f. 在给定输入x的情况下,本文提出的方案将使用f(|x|)作为分母近似常数,其中|x|是输入序列长度.
具体来说,本文使用预训练的Transformer模型对GLUE数据集中的若干条输入进行推理,给出测试数据集中不同的输入序列长度和Softmax注意力机制中分母均值的关系,如图5所示,横坐标的输入序列长度等于Softmax函数输入张量的维度,即为Softmax函数中分母求和时的项数,纵坐标表示对应的Softmax注意力机制中的分母均值.
图5表明分母均值并不随着输入序列长度的增加而线性增长. 本文使用幂函数$f_{{\mathrm{softmax}}}(\hat {{x}})= a{\hat {{x}}}^b$对其进行拟合,其中a=0.9923937666772096,b=0.333253864762922. 得到的拟合结果如图6所示.
得到的Softmax freeDiv Attention(简写为Soft_freeDivAttn)表达式如式(11)所示.
$ \begin{gathered}{{Soft\_freeDivAttn}}({\boldsymbol{Q}},{\boldsymbol{K}},{\boldsymbol{V}}) = {{Softma}}{{{x}}_{{\text{freeDiv}}}}\left( {\frac{{{\boldsymbol{Q}}{{\boldsymbol{K}}^{\text{T}}}}}{{\sqrt {{d}} }}} \right){\boldsymbol{V}}, \\ {{Softma}}{{{x}}_{{\text{freeDiv}}}}{\text{(}}{\boldsymbol{x}}{\text{) = }}\frac{{\left( {{{\text{e}}^{{x_0} - \max }}, … ,{{\text{e}}^{{x_{n - 1}} - \max }}} \right)}}{{{f_{{\text{softmax}}}}(|{\boldsymbol{x}}|)}}, \\ \end{gathered} $
其中,fsoftmax表示在Softmax Attention中从|x|到分母的近似常数的映射函数,其中|x|在进行Transformer推理时可获得,fsoftmax不依赖推理阶段的具体输入,可视为模型参数的一部分.
上述Softmax freeDiv Attention的思想及方法可以与其他注意力机制结合以获得更好的效果. 在MPCFormer中指出2Quad Attention在准确率和效率上都表现较为良好,因此可同样将Softmax freeDiv Attention的思想与2Quad Attention结合得到2Quad freeDiv Attention,同时需对Softmax freeDiv Attention做2项修改:
1)根据预训练Transformer模型在2Quad Attention注意力机制下测试不同输入序列长度时2Quad Attention注意力机制的分母数值,拟合得到二者之间的映射函数f2Quad;
2)将f2Quad与2Quad Attention结合,得到2Quad freeDiv Attention(简写为2Quad_freeDivAttn)的表达式如式(12)所示.
$ \begin{gathered}{{2Quad\_freeAttn}}({\boldsymbol{Q}},{{\boldsymbol{K}}},{\boldsymbol{V}}) = {{2Qua}}{{{d}}_{{\text{freeDiv}}}}\left( {\frac{{{\boldsymbol{Q}}{{K}^{\text{T}}}}}{{\sqrt {{d}} }}} \right){\boldsymbol{V}} ,\\ {{2Qua}}{{{d}}_{{\text{freeDiv}}}}{\text{(}}{\boldsymbol{x}}{\text{) = }}\frac{{\left( {{{({x_0} + c)}^2}, … ,{{({x_{n - 1}} + c)}^2}} \right)}}{{{f_{{{2{\mathrm{Quad}}}}}}(|{\boldsymbol{x}}|)}} .\\ \end{gathered} $
对于2Quad freeDiv Attention,按照上述1)中的方法得到的f2Quad为图7中拟合的幂函数,其中
a= 84.07373758071103,b=0.7174745255779887, c=5.
对于本文提出的2种注意力替换机制Softmax freeDiv Attention和2Quad freeDiv Attention,分别相比于Softmax Attention和2Quad Attention减少了1次除法和n次乘法的开销. 表1中给出各种注意力机制在ABY3框架下的计算延迟和速度提升倍数,本文提出的2种替换方案在延迟上均表现较好.
第5节中将会对这2种注意力机制以及以往工作中使用的注意力机制在效率和准确率上的表现进行更进一步的测试与比较.
对于激活函数GeLU的替换,以往方案中采取的较为高效的2种替换方式包括:将GeLU替换为ReLU、将GeLU替换为平方函数. 此外有的方案还会对GeLU用分段多项式进行精确近似,在准确率几乎不受影响的同时导致较低的计算效率. 由于在本文框架中后续会使用知识蒸馏技术提高模型的准确率,因此在权衡准确率和计算效率时可适当放宽对于准确率的要求从而追求更高的计算效率,不考虑对GeLU使用分段多项式进行精确近似的方案,而是选择ReLU和平方函数Quad这2种替换方法.
对于ReLU和Quad这2种激活函数对GeLU的替换,虽然Quad相比于ReLU少了1次比较运算,在计算上稍加高效,但由于同时进行了一些其他注意力机制的替换,使用Quad对GeLU进行替换相比于ReLU会导致准确率表现稍差,第5节中将对二者进行具体实验与说明.
通过上述分析,在注意力机制替换方面,本文选择使用Softmax freeDiv Attention或者2Quad freeDiv Attention这2种注意力机制,在对激活函数GeLU的替换方面,选择使用ReLU或者Quad来进行替换.
本节对MPC友好的Transformer转换框架在Transformer安全推理中应用时的效果进行实验与分析.
1)实验环境和MPC框架配置
实验统一在局域网下进行测试,操作系统为Ubuntu 20.04,处理器为Interl® CoreTM i9-10900X 3.70 GHz,知识蒸馏步骤使用GPU为NVIDIA Tesla P100 PCIe 16 GB.
在Transformer安全推理中,本实验借助于开源隐语框架SPU v0.6.0b0[32]来实现,这是一个通用的可用于隐私保护机器学习的框架. 在其中选择使用半诚实安全的ABY3协议[18]来实现Transformer安全推理.
2)模型结构和数据集
本文实验使用模型结构BERT-Base-Cased,使用数据集GLUE,针对的任务包括SST-2,STS-B,RTE. 在进行安全推理的延迟测试时,使用的输入序列长度为128. 由于Softmax函数的输入规模与输入序列长度的平方成正比,因此对于Softmax注意力机制的替换的加速效果会随着输入序列长度的增加而愈发显著. 为了后续与MPCFormer[21]对比,本文选择的输入序列长度与MPCFormer实验中相同,均为128.
3)知识蒸馏配置
本文实验在进行知识蒸馏时分为2步:第1步是对Transformer块进行蒸馏,使用学习率为5E−5,共迭代10次;第2步是对预测层进行蒸馏,使用学习率为1E−5,共迭代5次.
本节测试不同注意力机制替换方案与GeLU替换方案在准确率和效率上的表现,并将其与原始未进行替换的Transformer模型的基准方案进行对比. 选择的模型基础为BERT-Base-Cased,使用的数据集为GLUE,针对的任务为SST-2,STS-B,RTE,所使用的输入序列长度选择为128,主要对比的是各替换方案相比于基准值的相对值.
在注意力机制的替换方面,本文实验测试了以往方案中常用的多种注意力机制替换方法2ReLU Attention,2Quad Attention,Scaling Attention及本文提出的Softmax freeDiv Attention,2Quad freeDiv Attention. 在激活函数GeLU的替换方面,测试了将GeLU替换为ReLU和平方函数Quad两种方法. 同时将以上2种替换两两组合,并与基准方案(即未替换的Softmax注意力机制和激活函数GeLU)进行比较,得到的结果如表2所示.
从表2可以看出,在前4种注意力机制中,速度提升方面表现最好的是Scaling Attention,其与激活函数的Quad替换方法组合达到了2.26倍的速度提升,但其准确率大幅度下降. 而对于本文提出的2种注意力机制,可以看出对于2Quad freeDiv Attention + Quad的组合在速度提升方面与目前效率提升最明显的Scaling Attention相当,且在准确率方面表现良好,相比于原模型没有准确率的下降,因此在效率和准确率方面达到了目前所有方案中的最优.
表3给出了本文最终选择的表现最佳的2Quad freeDiv Attention + Quad的组合替换方案与原模型在准确率和安全推理速度上的对比. 可以看出,使用本文提出的MPC友好的Transformer转换框架转换后的模型在安全推理速度上达到了2.26倍的提升,且没有准确率损失,达到了目前已知方案中的最优.
本节测试不同注意力机制替换方案与GeLU替换方案在不同任务下的准确率和效率表现,以表明本文提出的MPC友好的Transformer转换框架的泛化性. 选择的模型基础为BERT-Base-Cased,使用的数据集为GLUE,针对GLUE数据集中的3类任务各选择一个具体任务:单句任务SST-2、相似性任务STS-B、释义任务RTE,所使用的输入序列长度选择为128.
在注意力机制的替换方面,本实验选择了Softmax Attention,2Quad Attention,Softmax freeDiv Attention,2Quad freeDiv Attention这4种替换方法. 在激活函数GeLU的替换方面,本文实验选择了Quad的替换方法. 得到的结果如表4所示,其中SST-2和RTE任务评估指标为准确率,STS-B任务评估指标为Pearson相关系数和Spearman相关系数的均值(后文简称为相关系数).
从表4可以看出,与基准值(Softmax Attention + GeLU)相比,本文提出的freeDiv Attention应用于不同任务上均有较好的准确率表现. 对于单句任务SST-2,freeDiv Attention至多达到了91.6%(上升0.1个百分点)的准确率;对于相似性任务STS-B,freeDiv Attention的相关系数至多达到了89.0%(上升0.3个百分点);对于释义任务RTE,freeDiv Attention至多达到了61.4%(下降1.8个百分点)的准确率,因此本文提出的freeDiv Attention在不同任务下具有一定的泛化性.
知识蒸馏过程是MPC友好的Transformer转换框架的重要组成部分,本文实验测试了知识蒸馏过程对于转换后模型的准确率影响,测试结果如表4所示.
由表4表明,对于SST-2任务,知识蒸馏过程至多可以提高42.4个百分点的准确率;对于STS-B任务,知识蒸馏过程至多可以提高90.2个百分点的相关系数;对于RTE任务,知识蒸馏过程至多可以提高12.6个百分点的准确率. 因此,知识蒸馏过程可以有效提高转换后模型的准确率.
本文提出的MPC友好的Transformer转换框架对不同任务的Transformer模型进行转换所需时间,即知识蒸馏步骤所需时间,结果如表5所示.
本节给出与本文使用同一安全推理框架(函数替换—知识蒸馏—安全推理)的MPCFormer方案[21]的对比结果,主要对比本文所提出的MPC友好的函数替换方案与MPCFormer中给出的表现最好的函数替换方案在同一数据集、同一任务上的准确率和延迟的表现. 对比结果如表6所示,基于本文的函数替换方案实现的Transformer安全推理在速度提升上优于MPCFormer.
本文提出了2种可用于Transformer的注意力机制freeDiv Attention注意力机制,并将其与激活函数GeLU的替换、知识蒸馏技术结合提出了一个MPC友好的Transformer转换框架,可以将预训练的Transformer模型转化为MPC友好的Transformer模型用于安全推理,在保证准确率不受影响的同时提高安全推理效率. 利用本文的MPC友好的Transformer转换框架将一个预训练的BERT-Base-Cased模型转化为MPC友好的模型之后,在GLUE数据集上针对SST-2任务可在保持模型准确率不降低的前提下,提高推理速度2.26倍. 在本文提出的2种freeDiv Attention注意力机制中,拟合分母均值时使用的是整个模型中所有层的均值,但观察到对于Transformer的不同层,甚至是多头注意力机制中的不同头,其注意力机制中的分母均值有不同分布,下一步可考虑对其进行细化划分,在一些较为复杂的任务上可能会有更好的效果.
作者贡献声明:刘伟欣和管晔玮提出了方案设计并撰写论文;霍嘉荣和丁元朝完成实验并撰写论文;郭华和李博提出指导意见并修改论文.

参考文献 引证文献
排序方式:
1
Hoffmann J, Borgeaud S, Mensch A, et al. An empirical analysis of compute-optimal large language model training[J]. Advances in Neural Information Processing Systems, 2022, 35: 30016−30030
2
Chan S, Santoro A, Lampinen A, et al. Data distributional properties drive emergent in-context learning in transformers[J]. Advances in Neural Information Processing Systems, 2022, 35: 18878−18891
3
Liu Ze, Lin Yutong, Cao Yue, et al. Swin transformer: Hierarchical vision transformer using shifted windows[C]//Proc of the IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE , 2021: 10012−10022
4
Liu Ze, Hu Han, Lin Yutong, et al. Swin transformer v2: Scaling up capacity and resolution[C]//Proc of the IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 12009−12019
5
Jawalkar N, Gupta K, Basu A, et al. Orca: FSS-based secure training with GPUs[J]. Cryptology ePrint Archive, 2023
6
Hao Meng, Li Hongwei, Chen Hanxiao, et al. Iron: Private inference on transformers[J]. Advances in Neural Information Processing Systems, 2022, 35: 15718−15731
7
Chen Tianyu, Bao Hangbo, Huang Shaohan, et al. THE-X: Privacy-preserving transformer inference with homomorphic encryption[C]//Findings of the Association for Computational Linguistics: ACL 2022. Stroudsburg, PA: ACL, 2022: 3510−3520
8
Zheng Mengxin, Lou Qian, Lei Jiang. Primer: Fast private transformer inference on encrypted data[J]. arXiv preprint, arXiv: 2303.13679, 2023
9
Gupta K, Jawalkar N, Mukherjee A, et al. Sigma: Secure gpt inference with function secret sharing[J]. Cryptology ePrint Archive, 2023
10
Juvekar C, Vaikuntanathan V, Chandrakasan A. GAZELLE: A low latency framework for secure neural network inference[C]//Proc of the 27th USENIX Conf on Security Symp. Berkeley, CA: USENIX Association, 2018: 1651−1669
11
Jiang Xiaoqian, Kim M, Lauter K, et al. Secure outsourced matrix computation and application to neural networks[C]//Proc of the 2018 ACM SIGSAC Conf on Computer and Communications Security. New York: Association for Computing Machinery, 2018: 1209−1222
12
Mohassel P, Zhang Y. SecureML: A system for scalable privacy-preserving machine learning[C]//Proc of 2017 IEEE Symp on Security and Privacy (SP). Piscataway, NJ: IEEE, 2017: 19−38
13
Huang Zhicong, Lu Wenjie, Hong Cheng, et al. Cheetah: Lean and fast secure two-party deep neural network inference[C]//Proc of the 31st USENIX Security Symp (USENIX Security 22). Berkeley, CA: USENIX Association, 2022: 809−826
14
Wang Ning, Xiao Xiaohui, Yang Yin, et al. Collecting and analyzing multidimensional data with local differential privacy[C]//Proc of the 2019 IEEE 35th Int Conf on Data Engineering (ICDE). Piscataway, NJ: IEEE, 2019: 638−649
15
Truong J B, Gallagher W, Guo Tian, et al. Memory-efficient deep learning inference in trusted execution environments[C]//Proc of the 2021 IEEE Int Conf on Cloud Engineering (IC2E). Piscataway, NJ: IEEE, 2021: 161−167
16
Akavia A, Leibovich M, Resheff Y S, et al. Privacy-preserving decision trees training and prediction[J]. ACM Transactions on Privacy and Security, 2022, 25(3): 1−30
17
Park S, Byun J, Lee J. Privacy-preserving fair learning of support vector machine with homomorphic encryption[C]//Proc of the ACM Web Conf 2022. New York: ACM, 2022: 3572−3583
18
Mohassel P, Rindal P. ABY3: A mixed protocol framework for machine learning[C]//Proc of the 2018 ACM SIGSAC Conf on Computer and Communications Security. New York: ACM, 2018: 35−52
19
Rathee D, Rathee M, Kumar N, et al. Cryptflow2: Practical 2-party secure inference[C]//Proc of the 2020 ACM SIGSAC Conf on Computer and Communications Security. New York: ACM, 2020: 325−342
20
Hou Xiaoyang, Liu Jian, Li Jingyu, et al. Ciphergpt: Secure two-party gpt inference[J]. Cryptology ePrint Archive, 2023
21
Li Dacheng, Shao Rulin, Wang Hongyi, et al. MPCFormer: Fast, performant and private transformer inference with MPC[J]. arXiv preprint, arXiv: 2211.01452, 2022
22
Zeng Wenxuan, Li Meng, Xiong Wenjie, et al. MPCViT: Searching for MPC-friendly vision transformer with heterogeneous attention[J]. arXiv preprint, arXiv: 2211.13955, 2022
23
Mishra P, Lehmkuhl R, Srinivasan A, et al. Delphi: A cryptographic inference system for neural networks[C]//Proc of the 29th USENIX Conf on Security Symp. Berkeley, CA: USENIX Association, 2020: 2505−2522
24
Wang Xiaolong, Girshick R, Gupta A, et al. Non-local neural networks[C]//Proc of the IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2018: 7794−7803
25
Wang Sinong, Li B Z, Khabsa M, et al. Linformer: Self-attention with linear complexity[J]. arXiv preprint, arXiv: 2006.04768, 2020
26
Rathee D, Rathee M, Goli R K K, et al. Sirnn: A math library for secure RNN inference[C]//Proc of 2021 IEEE Symp on Security and Privacy (SP). Piscataway, NJ: IEEE, 2021: 1003−1020
27
Dong Ye, Lu Wenjie, Zheng Yancheng, et al. Puma: Secure inference of LLaMA-7B in five minutes[J]. arXiv preprint, arXiv: 2307.12533, 2023
28
Akimoto Y, Fukuchi K, Akimoto Y, et al. Privformer: Privacy-preserving transformer with MPC[C]//Proc of 2023 IEEE 8th European Symp on Security and Privacy (EuroS&P). Piscataway, NJ: IEEE, 2023: 392−410
29
Wagh S, Tople S, Benhamouda F, et al. Falcon: Honest-majority maliciously secure framework for private deep learning[J]. arXiv preprint, arXiv: 2004.02229, 2020
30
Chou E, Beal J, Levy D, et al. Faster Cryptonets: Leveraging sparsity for real-world encrypted inference[J]. arXiv preprint, arXiv: 1811.09953, 2018
31
Hinton G, Vinyals O, Dean J. Distilling the knowledge in a neural network[J]. arXiv preprint , arXiv: 1503.02531, 2015
32
Ma Junming, Zheng Yancheng, Feng Jun, et al. SecretFlow-SPU: A performant and user-friendly framework for privacy-preserving machine learning[C]//Proc of 2023 USENIX Annual Technical Conf (USENIX ATC 23). Berkeley, CA: USENIX Associatioin, 2023: 17−33
2024年第61卷第5期
80
37
0
引用本文
BibTeX
文章信息
doi: 10.7544/issn1000-1239.202330966
  • 接收时间:2023-12-01
  • 首发时间:2026-11-17
  • 出版时间:2024-05-01
补充材料
相关文章
文章信息
作者
出版历史
  • 收稿日期:2023-12-01
  • 修回日期:2024-03-11
  • 录用日期:2024-03-11
基金
作者信息
    1北京航空航天大学网络空间安全学院 北京 100191
    2复杂关键软件环境全国重点实验室(北京航空航天大学) 北京 100878

通讯作者:

郭华()
参考文献
分享链接
https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202330966
分享至
全文二维码

扫描看全文

引用本文
BibTeX
本文的引用情况
2种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏