Article(id=1325489891596972958, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1325489890678420381, articleNumber=null, orderNo=null, doi=10.7544/issn1000-1239.202440804, pmid=null, cstr=null, oa=null, hot=null, price=null, onlineType=0, articleFormat=0, articleType=null, articleTypeStr=research-article, receivedDate=1729267200000, receivedDateStr=2024-10-19, revisedDate=1747584000000, revisedDateStr=2025-05-19, acceptedDate=1749571200000, acceptedDateStr=2025-06-11, onlineDate=1793951016567, onlineDateStr=2026-11-06, pubDate=1767196800000, pubDateStr=2026-01-01, doiRegisterDate=null, doiRegisterDateStr=null, onlineIssueDate=1793951016567, onlineIssueDateStr=2026-11-06, onlineJustAcceptDate=null, onlineJustAcceptDateStr=null, onlineFirstDate=null, onlineFirstDateStr=null, sourceXml=null, magXml=null, createTime=1793951016567, creator=13701087609, updateTime=1793951016567, updator=13701087609, issue=Issue{id=1325489890678420381, tenantId=1146029695717560320, journalId=1324759914146529478, year='2026', volume='63', issue='1', pageStart='1', pageEnd='272', issueExtLink='null', onlineDate='null', pubDate='1767196800000', pubDateStr='2026-01-01', beforeIssueId=null, nextIssueId=null, price=null, status=1, issueComplete=1, articleOrder=1, issueType=-1, specialIssue=null, createTime=1793951016348, creator='13701087609', updateTime=1793952522017, updator='13701087609', preIssue=null, nextIssue=null, articleTotal=null, ext={EN=IssueExt(id=1325496205974839692, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1325489890678420381, language=EN, specialIssueTitle=, coverIllustrator=null, specialIssueEditor=, specialIssueAbout=), CN=IssueExt(id=1325496205974839693, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1325489890678420381, language=CN, specialIssueTitle=, coverIllustrator=null, specialIssueEditor=, specialIssueAbout=)}, issueFiles=null, downloadFileDto=null}, startPage=162, endPage=175, ext={EN=ArticleExt(id=1325489892003820448, articleId=1325489891596972958, tenantId=1146029695717560320, journalId=1324759914146529478, language=EN, title=A Low-Precision Loss Quantification Method for Self-Attention Module, columnId=1325489891789910943, journalTitle=Journal of Computer Research and Development, columnName=Artificial Intelligence, runingTitle=null, highlight=null, articleAbstract=

With the rapid progress of deep learning technology and the continuous exploration of massive datasets, the self-attention module has been widely applied in various fields such as natural language processing, computer vision, and large language models. Although the self-attention module significantly improves the detection accuracy of deep learning models, its huge computational demand makes it particularly difficult to deploy on computing devices with limited computing power. Integer quantization, as one of the key technologies for deploying models on low-power computing chips, faces the problem of high precision loss caused by the structural characteristics of the self-attention module. To address this issue, a thorough analysis of the integer quantization error in the self-attention module is conducted, and two methods, pseudo-softmax vector quantization and block-wise pseudo-softmax vector quantization, are proposed. These two methods aim to significantly improve inference speed while effectively reducing the error caused by integer quantization by performing special integer quantization on the softmax vectors in the self-attention module. Experimental results show that compared with traditional direct quantization methods, the pseudo-softmax vector quantization method can reduce the quantization accuracy loss by 50%, while the block-wise pseudo-softmax vector quantization method can further reduce the accuracy loss by approximately 90%. These results fully demonstrate the effectiveness of the two quantization methods in reducing precision loss, providing strong support for the efficient deployment of the self-attention module on devices with limited computing power.

, authors=Delü Lin, Kun He*, authorsList=Delü Lin, Kun He, authorCompany=null, correspAuthors=Kun He, authorNote=

born in 1998. Master. His main research interests include high performance computing, model quantization, inference acceleration for large language models and autonomous driving models

born in 1972. PhD, professor. Distinguished member of CCF. Her main research interests include adversarial learning, deep representation learning, graph machine learning, and combinatorial optimization

, correspAuthorsNote=null, copyrightStatement=Copyright © 2026 Journal of Computer Research and Development. All rights reserved., copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=null, magXml=null, pdfUrl=null, pdf=null, pdfFileSize=null, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=null, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=null, mapNumber=null, fund=null), CN=ArticleExt(id=1325489900837024694, articleId=1325489891596972958, tenantId=1146029695717560320, journalId=1324759914146529478, language=CN, title=一种自注意力模块的低精度损失量化方法, columnId=1325489892104483745, journalTitle=计算机研究与发展, columnName=人工智能, runingTitle=null, highlight=null, articleAbstract=

随着深度学习技术的飞速进步和对海量数据集的持续发掘,自注意力模块在自然语言处理、计算机视觉以及大语言模型等多个领域得到了广泛应用。尽管自注意力模块显著提升了深度学习模型的检测精度,其巨大的计算需求却使得其在算力受限的计算设备上部署显得尤为困难。整数量化作为在低算力计算芯片中部署模型的关键技术之一,面临着由自注意力模块结构特点引起的较高精度损失问题。针对这个问题,对自注意力模块的整数量化误差进行了深入分析,提出了伪softmax向量量化方法和分块伪softmax向量量化方法。所提出方法通过对自注意力模块中的softmax向量进行特殊的整数量化,旨在显著提升推理速度的同时,有效降低整数量化带来的误差。实验结果表明,相比于传统的直接量化方法,伪softmax向量量化方法能够将量化精度损失降低50%,而分块伪softmax向量量化方法更是能将精度损失减少约90%。该结果充分证明了这2种量化方法在减少精度损失方面的有效性,为自注意力模块在算力受限设备上的高效部署提供了有力支持。

, authors=林德铝, 何琨*, authorsList=林德铝, 何琨, authorCompany=null, correspAuthors=何琨, authorNote=

1998年生。硕士。主要研究方向为高性能计算、模型量化、大语言模型以及自动驾驶模型的推理加速

1972年生。博士,教授。CCF杰出会员。主要研究方向为对抗学习、深度表示学习、图机器学习、组合优化

, correspAuthorsNote=
何琨()
, copyrightStatement=版权所有 © 2026 计算机研究与发展编辑部, copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=1AKvfteTI9FpUgdgzdYMAw==, magXml=d2WWQsk8h3NZFxzH27qZyg==, pdfUrl=null, pdf=oeHKNirJ04aCmCJUR1fymQ==, pdfFileSize=1612060, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=QHoyqgOTMcYxcSER68VTjg==, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=ByB8iJeqlAhgJNYand86cg==, mapNumber=null, fund=null)}, authors=[Author(id=1325489902040789947, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=derrylin@hust.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325489902472803261, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, authorId=1325489902040789947, language=EN, stringName=Delü Lin, firstName=Delü, middleName=null, lastName=Lin, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan 430074, bio={"content":"

born in 1998. Master. His main research interests include high performance computing, model quantization, inference acceleration for large language models and autonomous driving models

"}, bioImg=null, bioContent=

born in 1998. Master. His main research interests include high performance computing, model quantization, inference acceleration for large language models and autonomous driving models

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325489903835952062, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, authorId=1325489902040789947, language=CN, stringName=林德铝, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=华中科技大学计算机科学与技术学院 武汉 430074, bio={"img":"vU0f20rVdC/n8JmsFt4g5w==","content":"

1998年生。硕士。主要研究方向为高性能计算、模型量化、大语言模型以及自动驾驶模型的推理加速

"}, bioImg=vU0f20rVdC/n8JmsFt4g5w==, bioContent=

1998年生。硕士。主要研究方向为高性能计算、模型量化、大语言模型以及自动驾驶模型的推理加速

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325489901617165239, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, xref=null, ext=[AuthorCompanyExt(id=1325489901638136760, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, companyId=1325489901617165239, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan 430074), AuthorCompanyExt(id=1325489901650719673, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, companyId=1325489901617165239, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=华中科技大学计算机科学与技术学院 武汉 430074)])]), Author(id=1325489904184079297, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=brooklet60@hust.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=1, authorType=1, ext={EN=AuthorExt(id=1325489904528012227, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, authorId=1325489904184079297, language=EN, stringName=Kun He, firstName=Kun, middleName=null, lastName=He, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=*, address=School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan 430074, bio={"content":"

born in 1972. PhD, professor. Distinguished member of CCF. Her main research interests include adversarial learning, deep representation learning, graph machine learning, and combinatorial optimization

"}, bioImg=null, bioContent=

born in 1972. PhD, professor. Distinguished member of CCF. Her main research interests include adversarial learning, deep representation learning, graph machine learning, and combinatorial optimization

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325489904607704004, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, authorId=1325489904184079297, language=CN, stringName=何琨, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=*, address=华中科技大学计算机科学与技术学院 武汉 430074, bio={"img":"u0CjAogI/PNsL3DIXeyC0A==","content":"

1972年生。博士,教授。CCF杰出会员。主要研究方向为对抗学习、深度表示学习、图机器学习、组合优化

"}, bioImg=u0CjAogI/PNsL3DIXeyC0A==, bioContent=

1972年生。博士,教授。CCF杰出会员。主要研究方向为对抗学习、深度表示学习、图机器学习、组合优化

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325489901617165239, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, xref=null, ext=[AuthorCompanyExt(id=1325489901638136760, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, companyId=1325489901617165239, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan 430074), AuthorCompanyExt(id=1325489901650719673, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, companyId=1325489901617165239, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=华中科技大学计算机科学与技术学院 武汉 430074)])])], keywords=[Keyword(id=1325489904976802757, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, language=EN, orderNo=1, keyword=model quantization), Keyword(id=1325489908369994700, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, language=EN, orderNo=2, keyword=self-attention module), Keyword(id=1325489908449686477, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, language=EN, orderNo=3, keyword=low-precision loss), Keyword(id=1325489908508406734, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, language=EN, orderNo=4, keyword=inference acceleration), Keyword(id=1325489908709733327, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, language=EN, orderNo=5, keyword=divide and conquer), Keyword(id=1325489908793619408, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, language=CN, orderNo=1, keyword=模型量化), Keyword(id=1325489908864922577, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, language=CN, orderNo=2, keyword=自注意力模块), Keyword(id=1325489908940420050, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, language=CN, orderNo=3, keyword=低精度损失), Keyword(id=1325489909028500435, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, language=CN, orderNo=4, keyword=推理加速), Keyword(id=1325489909112386516, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, language=CN, orderNo=5, keyword=分治)], refs=[Reference(id=1325489912874677229, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=1, rfOrder=0, authorNames=null, journalName=null, refType=null, unstructuredReference=Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C]//Proc of the 31st Conf on Neural Information Processing Systems. Red Hook, NY: Curran Associates Inc, 2017: 5998−6008, articleTitle=null, refAbstract=null), Reference(id=1325489912971146222, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=2, rfOrder=1, authorNames=null, journalName=null, refType=null, unstructuredReference=廖兴滨, 秦小林, 张思齐, 等. 交互式机器翻译综述[J]. 计算机应用, 2023, 43(2): 329−334, articleTitle=null, refAbstract=null), Reference(id=1325489913034060783, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=2, rfOrder=2, authorNames=null, journalName=null, refType=null, unstructuredReference=Liao Xingbin, Qin Xiaolin, Zhang Siqi, et al. Review of interactive machine translation[J]. Journal of Computer Applications, 2023, 43(2): 329−334 (in Chinese), articleTitle=null, refAbstract=null), Reference(id=1325489913101169648, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=3, rfOrder=3, authorNames=null, journalName=null, refType=null, unstructuredReference=刘迪, 奚雪峰, 崔志明, 等. 抽取-生成式自动文本摘要技术研究综述[J]. 计算机技术与发展, 2023, 33(5): 1−8, articleTitle=null, refAbstract=null), Reference(id=1325489913168278513, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=3, rfOrder=4, authorNames=null, journalName=null, refType=null, unstructuredReference=Liu Di, Xi Xuefeng, Cui Zhiming, et al. Review of research on extractive-abstractive automatic text summarization technology[J]. Computer Technology and Development, 2023, 33(5): 1−8 (in Chinese), articleTitle=null, refAbstract=null), Reference(id=1325489913239581682, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=4, rfOrder=5, authorNames=null, journalName=null, refType=null, unstructuredReference=陈健鹏, 马建辉, 王怡君. 基于多轮交互的人机对话系统综述[J]. 南京信息工程大学学报, 2019, 11(3): 256−268, articleTitle=null, refAbstract=null), Reference(id=1325489913323467763, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=4, rfOrder=6, authorNames=null, journalName=null, refType=null, unstructuredReference=Chen Jianpeng, Ma Jianhui, Wang Yijun. A survey of human-computer dialogue system based on multiple-round interaction[J]. Journal of Nanjing University of Information Science & Technology, 2019, 11(3): 256−268 (in Chinese), articleTitle=null, refAbstract=null), Reference(id=1325489913394770932, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=5, rfOrder=7, authorNames=null, journalName=null, refType=null, unstructuredReference=Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16x16 words: Transformers for image recognition at scale[C/OL]//Proc of the 9th Int Conf on Learning Representations. Washington: ICLR, 2021[2025-03-29]. https://openreview.net/forum?id=YicbFdNTTy, articleTitle=null, refAbstract=null), Reference(id=1325489913457685493, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=6, rfOrder=8, authorNames=null, journalName=null, refType=null, unstructuredReference=Deng J, Dong W, Socher R, et al. ImageNet: A large-scale hierarchical image database[C]//Proc of the 22nd IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2009: 248−255, articleTitle=null, refAbstract=null), Reference(id=1325489913520600054, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=7, rfOrder=9, authorNames=null, journalName=null, refType=null, unstructuredReference=Radford A, Narasimhan K, Salimans T, et al. Improving language understanding with unsupervised learning[R]. San Francisco: OpenAI, 2018, articleTitle=null, refAbstract=null), Reference(id=1325489913583514615, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=8, rfOrder=10, authorNames=null, journalName=null, refType=null, unstructuredReference=Floridi L, Chiriatti M. GPT-3: Its nature, scope, limits, and consequences[J]. Minds and Machines, 2020, 30(4): 681−694, articleTitle=null, refAbstract=null), Reference(id=1325489913654817784, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=9, rfOrder=11, authorNames=null, journalName=null, refType=null, unstructuredReference=Touvron H, Lavril T, Izacard G, et al. LLaMA: Open and efficient foundation language models[J]. arXiv preprint, arXiv: 2302.13971, 2023, articleTitle=null, refAbstract=null), Reference(id=1325489913717732345, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=10, rfOrder=12, authorNames=null, journalName=null, refType=null, unstructuredReference=Ioffe S, Szegedy C. Batch normalization: Accelerating deep network training by reducing internal covariate shift[C]//Proc of the 32nd Int Conf on Machine Learning. Stroudsburg, PA: ACL, 2015: 448−456, articleTitle=null, refAbstract=null), Reference(id=1325489913772258298, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=11, rfOrder=13, authorNames=null, journalName=null, refType=null, unstructuredReference=Jacob B, Kligys S, Chen B, et al. Quantization and training of neural networks for efficient integer-arithmetic-only inference[C]//Proc of the 31st IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2018: 2704−2713, articleTitle=null, refAbstract=null), Reference(id=1325489913830978555, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=12, rfOrder=14, authorNames=null, journalName=null, refType=null, unstructuredReference=Sun Xiao, Choi J, Chen C Y, et al. Hybrid 8-bit floating point training and inference for deep neural networks[C]//Proc of the 33rd Conf on Neural Information Processing Systems. Red Hook, NY: Curran Associates Inc, 2019: 4901−4910, articleTitle=null, refAbstract=null), Reference(id=1325489913893893116, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=13, rfOrder=15, authorNames=null, journalName=null, refType=null, unstructuredReference=Lin Ji, Tang Jiaming, Tang Haotian, et al. AWQ: Activation-aware weight quantization for LLM compression and acceleration[C/OL]//Proc of the 7th Conf on Machine Learning and Systems. 2024[2025-03-29]. https://proceedings.mlsys.org/paper_files/paper/2024/file/42a452cbafa9dd64e9ba4aa95cc1ef21-Paper-Conference.pdf, articleTitle=null, refAbstract=null), Reference(id=1325489913986167805, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=14, rfOrder=16, authorNames=null, journalName=null, refType=null, unstructuredReference=Dettmers T, Pagnoni A, Holtzman A, et al. Qlora: Efficient finetuning of quantized LLMs[C]// Proc of the 37th Conf on Neural Information Processing Systems. Red Hook, NY: Curran Associates Inc, 2023: 10088−10115, articleTitle=null, refAbstract=null), Reference(id=1325489914061665278, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=15, rfOrder=17, authorNames=null, journalName=null, refType=null, unstructuredReference=Lin Yang, Zhang Tianyu, Sun Peiqin, et al. FQ-ViT: Post-training quantization for fully quantized vision transformer[C]//Proc of the 31st Int Joint Conf on Artificial Intelligence. Burlington: Morgan Kaufmann, 2022: 1173−1179, articleTitle=null, refAbstract=null), Reference(id=1325489914128774143, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=16, rfOrder=18, authorNames=null, journalName=null, refType=null, unstructuredReference=Yuan Zhihang, Xue Chenhao, Chen Yiqi, et al. PTQ4ViT: Post-training quantization for vision transformers with twin uniform quantization[C]//Proc of the 19th European Conf on Computer Vision. Berlin: Springer, 2022: 191−207, articleTitle=null, refAbstract=null), Reference(id=1325489914200077312, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=17, rfOrder=19, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Meiqi, Lu Siyuan, Zhu Danyang, et al. A high-speed and low-complexity architecture for softmax function in deep learning[C]//Proc of the 14th IEEE Asia Pacific Conf on Circuits and Systems. Piscataway, NJ: IEEE, 2018: 223−226, articleTitle=null, refAbstract=null), Reference(id=1325489914283962368, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=18, rfOrder=20, authorNames=null, journalName=null, refType=null, unstructuredReference=Vasyltsov I, Chang W. Efficient softmax approximation for deep neural networks with attention mechanism[J]. arXiv preprint, arXiv: 2111.10770, 2021, articleTitle=null, refAbstract=null), Reference(id=1325489914338488321, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=19, rfOrder=21, authorNames=null, journalName=null, refType=null, unstructuredReference=张强, 杨吉斌, 张雄伟, 等. CS-Softmax: 一种基于余弦相似性的Softmax损失函数[J]. 计算机研究与发展, 2022, 59(4): 936−949, articleTitle=null, refAbstract=null), Reference(id=1325489914409791490, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=19, rfOrder=22, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhang Qiang, Yang Jibin, Zhang Xiongwei, et al. CS-Softmax: A cosine similarity-based Softmax loss function[J]. Journal of Computer Research and Development, 2022, 59(4): 936−949 (in Chinese), articleTitle=null, refAbstract=null), Reference(id=1325489914569175043, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=20, rfOrder=23, authorNames=null, journalName=null, refType=null, unstructuredReference=Dao T, Fu D Y, Ermon S, et al. FlashAttention: Fast and memory-efficient exact attention with IO-awareness[C]//Proc of the 36th Conf on Neural Information Processing Systems. Red Hook, NY: Curran Associates Inc, 2022: 16344−16359, articleTitle=null, refAbstract=null), Reference(id=1325489914640478212, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=21, rfOrder=24, authorNames=null, journalName=null, refType=null, unstructuredReference=Li Zhiqi, Wang Wenhai, Li Hongyang, et al. BEVFormer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers[C]//Proc of the 19th European Conf on Computer Vision. Berlin: Springer, 2022: 1−18, articleTitle=null, refAbstract=null), Reference(id=1325489914715975685, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=22, rfOrder=25, authorNames=null, journalName=null, refType=null, unstructuredReference=Caesar H, Bankiti V, Lang A H, et al. NuScenes: A multimodal dataset for autonomous driving[C]//Proc of the 33rd IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2020: 11618−11628, articleTitle=null, refAbstract=null)], funds=null, companyList=[AuthorCompany(id=1325489901617165239, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, xref=null, ext=[AuthorCompanyExt(id=1325489901638136760, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, companyId=1325489901617165239, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan 430074), AuthorCompanyExt(id=1325489901650719673, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, companyId=1325489901617165239, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=华中科技大学计算机科学与技术学院 武汉 430074)])], figs=[ArticleFig(id=1325489909242409941, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, language=EN, label=Figure 1, caption=Original vector and softmax vector diagram, figureFileSmall=zsZXErXqzqg8C1mqY+ynkA==, figureFileBig=QHoyqgOTMcYxcSER68VTjg==, tableContent=null), ArticleFig(id=1325489909305324502, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, language=CN, label=图1, caption=原始向量及softmax向量图, figureFileSmall=zsZXErXqzqg8C1mqY+ynkA==, figureFileBig=QHoyqgOTMcYxcSER68VTjg==, tableContent=null), ArticleFig(id=1325489909527622615, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, language=EN, label=Figure 2, caption=softmax vector quantization error plot, figureFileSmall=62hXg0+Qh2KlbEjxwyJiCw==, figureFileBig=hVi1SkiiG0Vm/G0bhgWRpQ==, tableContent=null), ArticleFig(id=1325489909590537176, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, language=CN, label=图2, caption=softmax向量量化误差图, figureFileSmall=62hXg0+Qh2KlbEjxwyJiCw==, figureFileBig=hVi1SkiiG0Vm/G0bhgWRpQ==, tableContent=null), ArticleFig(id=1325489909649257433, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, language=EN, label=Figure 3, caption=Pseudo softmax vector quantization error plot, figureFileSmall=snTdbJ9MNQBPOBb3b/ahHg==, figureFileBig=7rtMNSXVkXuN1xWKSnIvfg==, tableContent=null), ArticleFig(id=1325489909712171994, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, language=CN, label=图3, caption=伪softmax向量量化误差图, figureFileSmall=snTdbJ9MNQBPOBb3b/ahHg==, figureFileBig=7rtMNSXVkXuN1xWKSnIvfg==, tableContent=null), ArticleFig(id=1325489909770892251, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, language=EN, label=Figure 4, caption=Quantization error plot of pseudo softmax vector information peak, figureFileSmall=fx5/v+smpsVy3OKsp0GWJQ==, figureFileBig=DxBxmLm3dRibIXK2bhd9Rw==, tableContent=null), ArticleFig(id=1325489909846389724, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, language=CN, label=图4, caption=伪softmax向量信息峰量化误差图, figureFileSmall=fx5/v+smpsVy3OKsp0GWJQ==, figureFileBig=DxBxmLm3dRibIXK2bhd9Rw==, tableContent=null), ArticleFig(id=1325489909959635933, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, language=EN, label=Figure 5, caption=Blocking diagram of multimodal pseudo softmax vector information peak, figureFileSmall=TUxJzmfDJ3PwVa1+vXCZOA==, figureFileBig=6twSDtgbui4geo7znrUjUw==, tableContent=null), ArticleFig(id=1325489910035133406, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, language=CN, label=图5, caption=多模态伪softmax向量信息峰分块图, figureFileSmall=TUxJzmfDJ3PwVa1+vXCZOA==, figureFileBig=6twSDtgbui4geo7znrUjUw==, tableContent=null), ArticleFig(id=1325489910173545439, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, language=EN, label=Figure 6, caption=Comparative diagram between pseudo softmax vector and block pseudo softmax vector, figureFileSmall=CwFq51AMALNbf7Nf+tz2zw==, figureFileBig=O87EY2jL8PZYZLvLm0iwHw==, tableContent=null), ArticleFig(id=1325489910236460000, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, language=CN, label=图6, caption=伪softmax向量与分块伪 softmax 向量对比图, figureFileSmall=CwFq51AMALNbf7Nf+tz2zw==, figureFileBig=O87EY2jL8PZYZLvLm0iwHw==, tableContent=null), ArticleFig(id=1325489910295180257, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, language=EN, label=Figure 7, caption=Error plot of block pseudo softmax vector quantization, figureFileSmall=MbyhjQ6yg7ChcOyGaMEJEg==, figureFileBig=OHKXnNHO8nnmnXJ3pdb9Rg==, tableContent=null), ArticleFig(id=1325489910353900514, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, language=CN, label=图7, caption=分块伪softmax向量量化误差图, figureFileSmall=MbyhjQ6yg7ChcOyGaMEJEg==, figureFileBig=OHKXnNHO8nnmnXJ3pdb9Rg==, tableContent=null), ArticleFig(id=1325489910421009379, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, language=EN, label=Figure 8, caption=Error plot of block pseudo softmax vector information peak quantization, figureFileSmall=ERMNxwwHhr+ypP7v7eW7RA==, figureFileBig=dMnvNTgy5iZ5Wu3KKmaUjg==, tableContent=null), ArticleFig(id=1325489910479729636, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, language=CN, label=图8, caption=分块伪softmax向量信息峰量化误差图, figureFileSmall=ERMNxwwHhr+ypP7v7eW7RA==, figureFileBig=dMnvNTgy5iZ5Wu3KKmaUjg==, tableContent=null), ArticleFig(id=1325489910555227109, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, language=EN, label=Table 1, caption=Experimental Results of Self-Attention Module Quantification, figureFileSmall=null, figureFileBig=null, tableContent=
方法数据类型耗时/ms平均绝对误差
官方闭源版本单精度8.830
半精度4.554.30×10−5
softmax直接量化8 b整数6.092.05×10−2
伪softmax向量量化8 b整数6.101.12×10−2
分块伪softmax向量量化
(WMMA指令)
8 b整数1.902.60×10−3
分块伪softmax向量量化
(MMA指令)
8 b整数1.162.60×10−3
), ArticleFig(id=1325489910622335974, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, language=CN, label=表1, caption=

自注意力模块量化实验结果

, figureFileSmall=null, figureFileBig=null, tableContent=
方法数据类型耗时/ms平均绝对误差
官方闭源版本单精度8.830
半精度4.554.30×10−5
softmax直接量化8 b整数6.092.05×10−2
伪softmax向量量化8 b整数6.101.12×10−2
分块伪softmax向量量化
(WMMA指令)
8 b整数1.902.60×10−3
分块伪softmax向量量化
(MMA指令)
8 b整数1.162.60×10−3
), ArticleFig(id=1325489910702027751, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, language=EN, label=Table 2, caption=Speed Measurement Result of Each Self-Attention Module, figureFileSmall=null, figureFileBig=null, tableContent=
方法数据类型模块耗时/ms
官方版本半精度矩阵乘(前)1.15
softmax2.56
矩阵乘(后)1.08
伪softmax
向量量化
8 b整数矩阵乘(前)2.33
softmax2.45
矩阵乘(后)1.15
量化处理0.17
), ArticleFig(id=1325489910790108136, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, language=CN, label=表2, caption=

各自注意力模块测速结果

, figureFileSmall=null, figureFileBig=null, tableContent=
方法数据类型模块耗时/ms
官方版本半精度矩阵乘(前)1.15
softmax2.56
矩阵乘(后)1.08
伪softmax
向量量化
8 b整数矩阵乘(前)2.33
softmax2.45
矩阵乘(后)1.15
量化处理0.17
), ArticleFig(id=1325489910932714473, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, language=EN, label=Table 3, caption=The Amount of Data Read and Write of Matrix Multiplication ( Front ), figureFileSmall=null, figureFileBig=null, tableContent=
计算类型输入类型输出类型读写量/MB
半精度半精度
(2 B)
半精度
(2 B)
读:33.55
写:536.87
8 b整数8 b整数
(1 B)
32 b整数
(4 B)
读:16.78
写:1073.74
), ArticleFig(id=1325489911016600554, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, language=CN, label=表3, caption=

矩阵乘(前)的数据读写量

, figureFileSmall=null, figureFileBig=null, tableContent=
计算类型输入类型输出类型读写量/MB
半精度半精度
(2 B)
半精度
(2 B)
读:33.55
写:536.87
8 b整数8 b整数
(1 B)
32 b整数
(4 B)
读:16.78
写:1073.74
), ArticleFig(id=1325489912677544939, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, language=EN, label=Table 4, caption=Impact of the softmax Quantization Method on Model Accuracy, figureFileSmall=null, figureFileBig=null, tableContent=
模型数据类型量化方法NDS
BEVFormer tiny浮点数35.3
8 b整数伪softmax量化法32.1
8 b整数分块伪softmax量化法35.4
BEVFormer small浮点数47.7
8 b整数伪softmax量化法45.4
8 b整数分块伪softmax量化法47.7
BEVFormer base浮点数51.5
8 b整数伪softmax量化法48.8
8 b整数分块伪softmax量化法51.4
), ArticleFig(id=1325489912757236716, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, language=CN, label=表4, caption=

softmax量化方法对模型精度的影响

, figureFileSmall=null, figureFileBig=null, tableContent=
模型数据类型量化方法NDS
BEVFormer tiny浮点数35.3
8 b整数伪softmax量化法32.1
8 b整数分块伪softmax量化法35.4
BEVFormer small浮点数47.7
8 b整数伪softmax量化法45.4
8 b整数分块伪softmax量化法47.7
BEVFormer base浮点数51.5
8 b整数伪softmax量化法48.8
8 b整数分块伪softmax量化法51.4
)], attaches=null, journal=Journal(id=1324759606947315908, delFlag=0, nameCn=计算机研究与发展, nameEn=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, issn=1000-1239, eissn=null, cn=11-1777/TP, coden=null, periodic=0, language=CN, oaType=null, ccby=null, superviseOffice=null, ownerOffice=null, pubOffice=null, editorOffice=null, officeType=null, aims=null, clcCode=null, officeProv=null, officeCity=null, officeAddr=null, officeZip=null, officeEmail=null, officePhone=null, editDirector=null, officeDirector=null, officeDirectorPhone=null, officeStaffNum=null, officeEmpNum=null, coverPicUrl=k2l8zgqwstuxxevFX2qxVA==, journalPrice=null, startedYear=null, abbrevIsoEn=Journal of Computer Research and Development, journalRemark=null, publicationField=null, createdTime=1793776903141, updatedTime=1793850698612, createdBy=18614031015, updatedBy=13701087609, firstLetterCn=J, firstLetterEn=J, subjectCode=Electronic Information, subjectName=null, subjectCodeEn=Electronic Information, subjectNameEn=null, picCn=k2l8zgqwstuxxevFX2qxVA==, picEn=r6k0Ry+RD+c/VXj5QarmDw==, jcr=null, cjcr=null, exts=[JournalExt(id=1325069127710286688, language=CN, name=计算机研究与发展, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698640, updatedTime=1793850698640, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""}), JournalExt(id=1325069127777395553, language=EN, name=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698656, updatedTime=1793850698656, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""})], databaseList=null, tenantJournalId=1324759914146529478, websiteList=[Website(id=1325070300139892771, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN, language=CN, createTime=1793850978169, createBy=18614031015, updateTime=1793851040259, updateBy=18614031015, name=计算机研究与发展-中文, tplId=1146099689490845704, title=计算机研究与发展, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070893751349360, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=articleTextType, value=kx, createTime=1793851119697, updateTime=1793851119697, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893730377837, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=banner, value=null, createTime=1793851119692, updateTime=1793851119692, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893772320883, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=grayFlag, value=0, createTime=1793851119702, updateTime=1793851119702, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893721989228, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851119690, updateTime=1793851119690, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893784903797, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=minRunFlag, value=0, createTime=1793851119705, updateTime=1793851119705, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893742960751, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic, createTime=1793851119695, updateTime=1793851119695, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893780709492, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=silenceFlag, value=0, createTime=1793851119704, updateTime=1793851119704, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893738766446, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_cn_619/, createTime=1793851119694, updateTime=1793851119694, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893755543665, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeColor, value=null, createTime=1793851119698, updateTime=1793851119698, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893763932274, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeStyle, value=null, createTime=1793851119700, updateTime=1793851119700, creator=18614031015, updator=18614031015)]), Website(id=1325070300580294698, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN, language=EN, createTime=1793850978274, createBy=18614031015, updateTime=1793851036638, updateBy=18614031015, name=计算机研究与发展-英文, tplId=1146101810881728533, title=Journal of Computer Research and Development, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070925288321148, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=articleTextType, value=kx, createTime=1793851127216, updateTime=1793851127216, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925263155321, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=banner, value=null, createTime=1793851127210, updateTime=1793851127210, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925309292671, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=grayFlag, value=0, createTime=1793851127221, updateTime=1793851127221, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925254766712, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851127208, updateTime=1793851127208, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925326069889, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=minRunFlag, value=0, createTime=1793851127225, updateTime=1793851127225, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925279932539, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic, createTime=1793851127214, updateTime=1793851127214, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925317681280, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=silenceFlag, value=0, createTime=1793851127223, updateTime=1793851127223, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925271543930, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_en_623/, createTime=1793851127212, updateTime=1793851127212, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925292515453, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeColor, value=null, createTime=1793851127217, updateTime=1793851127217, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925300904062, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeStyle, value=null, createTime=1793851127219, updateTime=1793851127219, creator=18614031015, updator=18614031015)])], journalTitle=计算机研究与发展, weixinUrl=null, journalUrl=https://crad.ict.ac.cn/, iacademicId=null, status=1, seqNo=null, journalTitleEn=Journal of Computer Research and Development, journalPhotoCn=k2l8zgqwstuxxevFX2qxVA==, journalPhotoEn=r6k0Ry+RD+c/VXj5QarmDw==, journalFirstLetter=J, journalRecommend=null, journalNew=null, journalCollection=null, jcrJf=null, cjcrJf=null, jcrJfStr=null, cjcrJfStr=null, submissionFirstDecision=null, sciSubjectClassification=null, casSubjectClassification=null, citeScore=null, totalCitationFrequency=null, icpCode=null, psCode=null, advertisingLicenseCode=null, copyrightInformation=null, country=null, option=, provinceCode=null, provinceName=null, collectFlag=false, interPubPlatform=, interPubPlatformUrl=null), detailUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202440804, detailUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202440804, pdfUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/PDF/10.7544/issn1000-1239.202440804, pdfUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/PDF/10.7544/issn1000-1239.202440804, aliStartDate=0, aliEndDate=0, collectionFlag=false, citedCount=null, citedUrl=null, previewStatus=0, delFlag=0, hasFullText=1, orderTime=1767196800000, fullTextJson=null, articleText=null, reference=null)
收藏切换
一种自注意力模块的低精度损失量化方法
林德铝 , 何琨 *
计算机研究与发展 | 人工智能 2026,63(1): 162-175
收起
收藏切换
计算机研究与发展 |人工智能 2026 , 63 (1) : 162 -175
一种自注意力模块的低精度损失量化方法
全屏
[Author(id=1325489902040789947, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=derrylin@hust.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325489902472803261, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, authorId=1325489902040789947, language=EN, stringName=Delü Lin, firstName=Delü, middleName=null, lastName=Lin, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan 430074, bio={"content":"

born in 1998. Master. His main research interests include high performance computing, model quantization, inference acceleration for large language models and autonomous driving models

"}, bioImg=null, bioContent=

born in 1998. Master. His main research interests include high performance computing, model quantization, inference acceleration for large language models and autonomous driving models

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325489903835952062, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, authorId=1325489902040789947, language=CN, stringName=林德铝, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=华中科技大学计算机科学与技术学院 武汉 430074, bio={"img":"vU0f20rVdC/n8JmsFt4g5w==","content":"

1998年生。硕士。主要研究方向为高性能计算、模型量化、大语言模型以及自动驾驶模型的推理加速

"}, bioImg=vU0f20rVdC/n8JmsFt4g5w==, bioContent=

1998年生。硕士。主要研究方向为高性能计算、模型量化、大语言模型以及自动驾驶模型的推理加速

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325489901617165239, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, xref=null, ext=[AuthorCompanyExt(id=1325489901638136760, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, companyId=1325489901617165239, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan 430074), AuthorCompanyExt(id=1325489901650719673, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, companyId=1325489901617165239, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=华中科技大学计算机科学与技术学院 武汉 430074)])]), Author(id=1325489904184079297, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=brooklet60@hust.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=1, authorType=1, ext={EN=AuthorExt(id=1325489904528012227, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, authorId=1325489904184079297, language=EN, stringName=Kun He, firstName=Kun, middleName=null, lastName=He, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=*, address=School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan 430074, bio={"content":"

born in 1972. PhD, professor. Distinguished member of CCF. Her main research interests include adversarial learning, deep representation learning, graph machine learning, and combinatorial optimization

"}, bioImg=null, bioContent=

born in 1972. PhD, professor. Distinguished member of CCF. Her main research interests include adversarial learning, deep representation learning, graph machine learning, and combinatorial optimization

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325489904607704004, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, authorId=1325489904184079297, language=CN, stringName=何琨, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=*, address=华中科技大学计算机科学与技术学院 武汉 430074, bio={"img":"u0CjAogI/PNsL3DIXeyC0A==","content":"

1972年生。博士,教授。CCF杰出会员。主要研究方向为对抗学习、深度表示学习、图机器学习、组合优化

"}, bioImg=u0CjAogI/PNsL3DIXeyC0A==, bioContent=

1972年生。博士,教授。CCF杰出会员。主要研究方向为对抗学习、深度表示学习、图机器学习、组合优化

, aboutCorrespAuthor=null)}, companyList=[AuthorCompany(id=1325489901617165239, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, xref=null, ext=[AuthorCompanyExt(id=1325489901638136760, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, companyId=1325489901617165239, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan 430074), AuthorCompanyExt(id=1325489901650719673, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325489891596972958, companyId=1325489901617165239, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=华中科技大学计算机科学与技术学院 武汉 430074)])])]
林德铝 , 何琨*
作者信息
  • 华中科技大学计算机科学与技术学院 武汉 430074
通讯作者:
作者简介:

1998年生。硕士。主要研究方向为高性能计算、模型量化、大语言模型以及自动驾驶模型的推理加速

1972年生。博士,教授。CCF杰出会员。主要研究方向为对抗学习、深度表示学习、图机器学习、组合优化

A Low-Precision Loss Quantification Method for Self-Attention Module
Delü Lin , Kun He*
Affiliations
  • School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan 430074
  • born in 1998. Master. His main research interests include high performance computing, model quantization, inference acceleration for large language models and autonomous driving models

    born in 1972. PhD, professor. Distinguished member of CCF. Her main research interests include adversarial learning, deep representation learning, graph machine learning, and combinatorial optimization

出版时间: 2026-01-01 doi: 10.7544/issn1000-1239.202440804
文章导航
收藏切换

随着深度学习技术的飞速进步和对海量数据集的持续发掘,自注意力模块在自然语言处理、计算机视觉以及大语言模型等多个领域得到了广泛应用。尽管自注意力模块显著提升了深度学习模型的检测精度,其巨大的计算需求却使得其在算力受限的计算设备上部署显得尤为困难。整数量化作为在低算力计算芯片中部署模型的关键技术之一,面临着由自注意力模块结构特点引起的较高精度损失问题。针对这个问题,对自注意力模块的整数量化误差进行了深入分析,提出了伪softmax向量量化方法和分块伪softmax向量量化方法。所提出方法通过对自注意力模块中的softmax向量进行特殊的整数量化,旨在显著提升推理速度的同时,有效降低整数量化带来的误差。实验结果表明,相比于传统的直接量化方法,伪softmax向量量化方法能够将量化精度损失降低50%,而分块伪softmax向量量化方法更是能将精度损失减少约90%。该结果充分证明了这2种量化方法在减少精度损失方面的有效性,为自注意力模块在算力受限设备上的高效部署提供了有力支持。

模型量化  /  自注意力模块  /  低精度损失  /  推理加速  /  分治

With the rapid progress of deep learning technology and the continuous exploration of massive datasets, the self-attention module has been widely applied in various fields such as natural language processing, computer vision, and large language models. Although the self-attention module significantly improves the detection accuracy of deep learning models, its huge computational demand makes it particularly difficult to deploy on computing devices with limited computing power. Integer quantization, as one of the key technologies for deploying models on low-power computing chips, faces the problem of high precision loss caused by the structural characteristics of the self-attention module. To address this issue, a thorough analysis of the integer quantization error in the self-attention module is conducted, and two methods, pseudo-softmax vector quantization and block-wise pseudo-softmax vector quantization, are proposed. These two methods aim to significantly improve inference speed while effectively reducing the error caused by integer quantization by performing special integer quantization on the softmax vectors in the self-attention module. Experimental results show that compared with traditional direct quantization methods, the pseudo-softmax vector quantization method can reduce the quantization accuracy loss by 50%, while the block-wise pseudo-softmax vector quantization method can further reduce the accuracy loss by approximately 90%. These results fully demonstrate the effectiveness of the two quantization methods in reducing precision loss, providing strong support for the efficient deployment of the self-attention module on devices with limited computing power.

model quantization  /  self-attention module  /  low-precision loss  /  inference acceleration  /  divide and conquer
林德铝, 何琨. 一种自注意力模块的低精度损失量化方法. 计算机研究与发展, 2026 , 63 (1) : 162 -175 . DOI: 10.7544/issn1000-1239.202440804
Delü Lin, Kun He. A Low-Precision Loss Quantification Method for Self-Attention Module[J]. Journal of Computer Research and Development, 2026 , 63 (1) : 162 -175 . DOI: 10.7544/issn1000-1239.202440804
随着深度学习技术的迅猛发展,基于神经网络设计的模型在学术界和工业界均得到了广泛的关注和应用。为了解决传统深度学习模型在处理序列数据时的不足,基于自注意力(self-attention)设计的Transformer模型[1]应运而生,并迅速在自然语言处理任务如机器翻译[2]、文本摘要[3]、人机对话[4]等领域中崭露头角。随后,Dosovitskiy等人[5]将Transformer模型的创新理念引入计算机视觉领域,提出了具有划时代意义的ViT(vision Transformer)模型。ViT模型在ImageNet[6]竞赛中一举超越了传统的卷积神经网络,预示着它将成为未来几年计算机视觉领域的研究焦点。同时,大语言模型的发展也离不开Transformer结构的支撑,主流的大语言模型如GPT[7-8],LLaMA[9] 均是以Transformer为核心进行设计的。
然而,随着Transformer结构的广泛应用,如何在工业界实现其轻量化部署成为了一个亟待解决的问题。在工业界的模型部署实践中,模型整数量化已成为主流的轻量化部署手段。这种方法通过线性映射函数将模型的浮点数运算转换为低比特整数运算,从而在边缘计算设备上进行高效的低比特整数线性运算,显著提升了模型的部署效率。在卷积神经网络和全连接线性层中,通过网络融合的方式,将卷积神经网络或全连接层与批归一化[10]和ReLU激活函数合并为一个整体,再对整体进行低比特整数量化运算,从而大幅提高基于卷积神经网络和全连接层设计的模型推理效率。
对于Transformer模型而言,由于其包含了大量的自注意力模块,这些模块的量化过程往往面临较大的挑战。自注意力模块通过查询(query)矩阵Q和键(key)矩阵K的相乘,并经过softmax函数进行归一化,再与值(value)矩阵V相乘得到加权输出。由于softmax函数的非线性特性,自注意力模块的量化往往会导致较大的精度损失。在工业界的Transformer模型部署场景中,自注意力模块部分通常仍采用浮点数进行计算,这在一定程度上限制了整体模型的量化率和部署计算效率。因此,开展对自注意力模块的低精度损失整数量化方法的研究显得尤为重要。
针对上述问题,本文深入剖析了自注意力模块量化误差的成因,并提出了一种针对性的高效量化方法。这种方法能够显著提升自注意力模块的量化精度,从而优化模型的部署效率。此外,本文还将这种自注意力模块的量化方法在英伟达的图形处理器GPU中进行了实现,并通过一系列严谨的算子实验验证了所提出方法的有效性,为Transformer模型的轻量化部署提供了有力的理论和技术支持。
本文的主要贡献包括2个方面:
1)深入分析并指出了自注意力模块的直接量化存在的误差问题,提出了伪softmax向量量化和分块伪softmax向量量化方法;
2)在英伟达的图形处理器中实现了这些量化方法的算子,在平均绝对误差和耗时2个指标上验证了所提量化方法的有效性。
模型的整数量化是一种普遍应用的轻量化部署技术,在模型推理过程中扮演着至关重要的角色。通过采用较低的数值精度,如8 b整数(integer 8,INT8)进行推理,能够显著降低显存占用和延迟,进而提升数据吞吐率。这一技术主要是通过将原始的深度神经网络高精度浮点数据类型转换为低比特定点数据类型实现,常见的低比特定点数据类型有8 b整数和4 b整数等。低比特整数量化推理有2个主要优点:一是可以更快地通过内存层次结构移动数据,以最大限度地利用计算资源;二是在较低的数值精度下硬件可以实现更高的每秒运算次数,因为这些乘法器需要更少的硅面积和功率。
8 b整数量化推理作为定点数量化推理的代表性技术,在深度学习模型推理中广泛应用。8 b整数量化推理是指使用8 b整数格式进行深度学习模型推理,以实现更高的计算效率和能效。随着深度学习算法和模型应用范围的日益扩大,对高效、节能的推理计算需求也急剧增长,这使得8 b整数推理技术得到了广泛的研究和应用。
量化通常针对线性算子设计,Jacob等人[11]提出了8 b整数量化方法,使深度学习神经网络的线性算子仅依赖整型数据进行计算,不仅将存储空间缩减至原来的1/4,还显著提升了推理效率。Sun等人[12]则尝试使用8 b浮点数(floating point 8,FP8)进行量化,以减轻量化带来的推理精度损失,但这通常需要硬件架构的支持。另外,Lin等人[13]提出的AWQ量化方法,通过对线性层权重矩阵进行分组量化,实现了以4 b整数进行存储和计算。Dettmers等人[14]则提出了一种全新的4 b数据格式NormalFloat,它更好地适应了高斯分布数据,有效降低了模型量化带来的精度损失。
尽管线性算子的量化取得了显著成果,但直接将其应用于包含自注意机制的模型往往会导致精度显著下降。因此,针对这些模块,需要研究如何在提升速度的同时保证精度。例如,Lin等人[15]提出的FQ-ViT模型使用了Log-Int-Softmax模块,以替代自注意力模块中的softmax模块,使其能够进行8 b整数推理。然而,这种方法改变了模型结构,需要重新进行微调训练,可能导致模型整体精度下降。Yuan等人[16]提出了一种基于孪生均匀量化(twin uniform quantization)方法的模型PTQ4ViT,将视觉自注意力模块中的激活值分为2个范围,并对每个范围进行单独的均匀量化。这种量化方式相对于线性量化而言,计算复杂度更高,可能降低8 b整数量化的计算效率。
从计算硬件设备的角度,Wang等人[17]利用数学变换,将softmax整数量化的指数和对数计算转换为有限输入范围的计算以及一系列简单的计算,比如常数乘法运算、移位运算、加法运算等。但是该方法需要硬件结构支持才能发挥最大作用,在当下主流的边缘计算推理芯片中不具备通用性。Vasyltsov等人[18]基于分段常数近似和查找表(look-up table,LUT)提出了一种高效的softmax计算方法,利用了整数量化的精度,可以与大部分传统的硬件加速器兼容。但是该方法的资源消耗以存储为主,需要利用高速而昂贵有限的片上存储。
在算法层面上,张强等人[19]提出了一种基于余弦相似性的softmax损失函数CS-Softmax,该函数在分类任务中表现出了优异的性能。但是其依旧是采用浮点数进行计算,无法提升模型推理速度。
针对上述挑战,本文提出了一种针对自注意力模块的量化方法。它可以在不改变softmax算法和线性量化方式的前提下,在通用的图形处理器推理设备上支持8 b量化。同时,本文以图1所示的常见的softmax向量分布为例,对其进行量化误差分析,论证所提出的量化方法可以有效控制softmax向量的量化精度损失。
通常自注意力模块算子的输入为查询矩阵Q、键矩阵K和值矩阵V,算子计算主要包含2个批量矩阵乘以及1个softmax函数,其计算方式如式(1)所示。
$ \begin{split}&\quad\quad\quad\quad{\boldsymbol{Q}}\in {\mathbb{R}}^{B\times {L}_{\boldsymbol{Q}}\times d}\text{,}\\ &\quad\quad\quad{\boldsymbol{K}}\text{,}{\boldsymbol{V}}\in {\mathbb{R}}^{B\times {L}_{\boldsymbol{KV}}\times d}\text{,}\\&\quad softmax\left({\boldsymbol{x}}\right)=\frac{\mathrm{exp}\left({\boldsymbol{x}}\right)}{{\displaystyle \sum \mathrm{exp}}({\boldsymbol{x}})}\text{,}\\&{\boldsymbol{S}}=softmax\left(mm\left(\frac{{\boldsymbol{Q}}}{\sqrt{d}}\text{,}{{\boldsymbol{K}}}^{\text{T}}\right)\right)\text{,}\\&attention({\boldsymbol{Q}}\text{,}{\boldsymbol{K}}\text{,}{\boldsymbol{V}})=mm\left({\boldsymbol{S}}\text{,}{\boldsymbol{V}}\right)\;。\end{split} $
其中Q,K,V分别代表查询矩阵、键矩阵和值矩阵;B为数据的批大小,$ {L}_{\boldsymbol{Q}} $和$ {L}_{\boldsymbol{K}\boldsymbol{V}} $分别为Q以及K或V的序列长度,即键矩阵和值矩阵的序列长度均为$ {L}_{\boldsymbol{K}\boldsymbol{V}} $;d为每个序列的通道大小,$mm(\cdot) $为批量矩阵乘。
在自注意力模块的量化推理实践中,考虑到softmax的输出矩阵范围固定在[0, 1]之间,且理论上该范围内的任意值都是可能达到的。因为无符号8 b整数的范围为[0, 255],所以softmax输出向量的缩放系数通常被固定为1/255,这种固定值的做法是造成自注意力模块量化误差的主要原因。为了解决这个问题,工业界常见的方案是放弃对自注意力模块中最后1个矩阵乘法的量化,即只量化第1个矩阵乘法运算。该做法虽然能够减少量化误差,但却降低了整体的量化率,无法充分发挥量化带来的性能提升和资源优势。通常,为了避免指数运算的溢出,softmax函数在计算机内的计算方式如式(2)所示。
$ \begin{split}&\quad\quad\quad\quad\quad\quad m\left({\boldsymbol{x}}\right)=\underset{i}{{{\mathrm{max}}}}{x}_{i}\text{,}\\&f\left({\boldsymbol{x}}\right)=\left[{\exp}({x}_{1}-m({\boldsymbol{x}}))\text{,}… \text{,}\mathrm{exp}({x}_{n}-m({\boldsymbol{x}}))\right]\text{,}\\&\quad\quad\quad\quad\quad\quad l\left({\boldsymbol{x}}\right)={\displaystyle \sum f\left({\boldsymbol{x}}\right)}\text{,}\\&\quad\quad\quad\quad\quad softmax\left({\boldsymbol{x}}\right)=\frac{f\left({\boldsymbol{x}}\right)}{l\left({\boldsymbol{x}}\right)}\;,\end{split} $
即在计算指数运算前会先减掉向量的最大值,使得指数运算输出范围在[0, 1],然后再除以向量和,这与原始的softmax计算等价。
然而,由于softmax的输出特性,其指数运算无法只用定点数进行计算,只能使用浮点数,需要对其输出进行量化方可在之后的计算使用定点数运算。其输出范围为[0, 1],若直接将该范围用作计算缩放系数,则缩放系数为1/255。现考虑对一个1024维的向量直接进行计算并量化输出,显然,量化输出的向量取值范围为[0, 255]的整数,即将浮点数向量除以量化缩放系数再进行取整,如式(3)所示。
$ {s}_{\text{s}}=\frac{1}{255}\text{,}{{\boldsymbol{x}}}_{\text{INT8}}^{\text{s}}=\left\lfloor \frac{{{\boldsymbol{x}}}_{\text{f}}^{\text{s}}}{{s}_{\text{s}}} \right\rceil,$
其中ss为softmax输出向量的8 b量化缩放系数, $ {\boldsymbol{x}}_{\mathrm{f}}^{\mathrm{s}} $和$ {\boldsymbol{x}}_{\mathrm{I}\mathrm{N}\mathrm{T}8}^{\mathrm{s}} $分别为softmax输出的浮点数向量和8 b整数量化向量,$ \lfloor \cdot \rceil $为四舍五入取整。
现考虑对几种常见的softmax分布进行误差分析,以论证不同量化方式对softmax量化误差的影响。如图1所示,图1(a)~(d)为原始1 024维向量,包括均匀、单峰、双峰及多峰分布形态的向量;而图1(e)~(h)为对应向量经过softmax计算后的向量。其中,横坐标为对应向量的位置索引,纵坐标为向量值。
若直接对该softmax向量进行8 b整数量化,将其输出与量化前的softmax向量组合可得到量化误差图,如图2所示。其中,横坐标为1024维向量的索引,纵坐标为向量对应位置的值,实线为softmax浮点数输出的分布,虚线为softmax量化输出的分布,填充区域代表这2种分布之间的误差,$ {\delta }_{\mathrm{m}\mathrm{e}\mathrm{a}\mathrm{n}} $,$ {\delta }_{\mathrm{m}\mathrm{a}\mathrm{x}} $,$ {\delta }_{\mathrm{m}\mathrm{i}\mathrm{n}} $分别为平均绝对误差、最大绝对误差和最小绝对误差,其由式(4)计算得到。
$ \begin{aligned}&{\delta }_{\text{mean}}=\frac{{\displaystyle \sum _{i=1}^{n}\left|{x}_{\text{f}}^{\text{s}}{}_{\left(i\right)}-{s}_{\text{s}}{x}_{\text{INT8}}^{\text{s}}{}_{\left(i\right)}\right|}}{n}\text{,}\\&{\delta }_{\text{max}}=\underset{1\le i\le n}{\mathrm{max}}\left|{x}_{\text{f}}^{\text{s}}{}_{\left(i\right)}-{s}_{\text{s}}{x}_{\text{INT8}}^{\text{s}}{}_{\left(i\right)}\right|\text{,}\\ &{\delta }_{\text{min}}=\underset{1\le i\le n}{\mathrm{min}}\left|{x}_{\text{f}}^{\text{s}}{}_{\left(i\right)}-{s}_{\text{s}}{x}_{\text{INT8}}^{\text{s}}{}_{\left(i\right)}\right|,\end{aligned} $
其中$ {{x}_{\mathrm{f}}^{\mathrm{s}}}_{\left(i\right)} $和$ {{x}_{\mathrm{I}\mathrm{N}\mathrm{T}8}^{\mathrm{s}}}_{\left(i\right)} $分别为softmax浮点数向量$ {\boldsymbol{x}}_{\mathrm{f}}^{\mathrm{s}} $和量化整数向量$ {\boldsymbol{x}}_{\mathrm{I}\mathrm{N}\mathrm{T}8}^{\mathrm{s}} $的第i个值,n为向量长度。
由此可见,不论何种分布,量化后的向量会出现大量的0,导致巨大的误差。这是因为softmax的输出和为1,在较大的向量中会有大量的值远小于1,然而缩放系数依旧使用最大绝对值为1来计算,从而导致误差增大。
通过观察softmax的计算过程,可以发现一个有趣的特性,即在计算softmax之前,通常会先将输入向量中的每个元素减去该向量的最大值。这样做的目的是防止在指数运算过程中发生数值溢出,并确保结果向量中的所有值都小于等于0,且至少有1个值为“0”(即原本的最大值)。接着,对这个经过调整的向量进行指数运算,得到的输出向量范围仍然在[0, 1]之间,并且至少有1个元素的值为“1”。这个向量与softmax的输出非常相似,只是没有进行最后的归一化步骤,因此我们可以称之为“伪softmax向量”。
伪softmax向量是一个非常适合做定点数量化的向量,因为其正好解决了原softmax向量有大量值远小于1的问题,伪softmax向量必存在且至少含有1个最大值“1”,为量化提供了良好的条件。同样地,对均匀、单峰、双峰以及多峰的输入进行伪softmax计算,并对其输出进行量化,分析定点数误差,如图3所示。
与原来的softmax向量输出相比,伪softmax向量未归一,误差会随之放大。因此,为了与原来的误差估计对齐,对于伪softmax向量的误差估计$ {\delta }_{\mathrm{m}\mathrm{e}\mathrm{a}\mathrm{n}} $,$ {\delta }_{\mathrm{m}\mathrm{a}\mathrm{x}} $,$ {\delta }_{\mathrm{m}\mathrm{i}\mathrm{n}} $由式(5)计算得到。
$ \begin{aligned}&{s}_{\text{f}}^{\text{ws}}={\displaystyle\sum _{i}{x}^{\text{ws}}_{{\text{f}}{\left(i\right)}}}\text{,}{s}_{\text{INT8}}^{\text{ws}}={\displaystyle\sum _{i}{x}^{\text{ws}}_{{\text{INT8}}{\left(i\right)}}}\text{,}\\&{\delta }_{\text{mean}}=\dfrac{{\displaystyle \sum _{i=1}^{n}\left|\dfrac{{x}^{\text{ws}}_{{\text{f}}{\left(i\right)}}}{{s}_{\text{f}}^{\text{ws}}}-\dfrac{{x}^{\text{ws}}_{{\text{INT8}}{\left(i\right)}}}{{s}_{\text{INT8}}^{\text{ws}}}\right|}}{n}\text{,}\\&{\delta }_{\text{max}}=\underset{1\le i\le n}{\mathrm{max}}\left|\dfrac{{x}^{\text{ws}}_{{\text{f}}{\left(i\right)}}}{{s}_{\text{f}}^{\text{ws}}}-\dfrac{{x}^{\text{ws}}_{{\text{INT8}}{\left(i\right)}}}{{s}_{\text{INT8}}^{\text{ws}}}\right|\text{,}\\&{\delta }_{\text{min}}=\underset{1\le i\le n}{\mathrm{min}}\left|\dfrac{{x}^{\text{ws}}_{{\text{f}}{\left(i\right)}}}{{s}_{\text{f}}^{\text{ws}}}-\dfrac{{x}^{{\mathrm{ws}}}_{{{\mathrm{INT}}8}{\left(i\right)}}}{{s}_{\text{INT8}}^{\text{ws}}}\right|,\end{aligned} $
其中$ {{x}_{\mathrm{f}}^{\mathrm{w}\mathrm{s}}}_{\left(i\right)} $和$ {{x}_{\mathrm{I}\mathrm{N}\mathrm{T}8}^{\mathrm{w}\mathrm{s}}}_{\left(i\right)} $分别为伪softmax浮点数向量$ {\boldsymbol{x}}_{\mathrm{f}}^{\mathrm{w}\mathrm{s}} $和量化整数向量$ {\boldsymbol{x}}_{\mathrm{I}\mathrm{N}\mathrm{T}8}^{\mathrm{w}\mathrm{s}} $的第i个值,n为向量长度。由图3可见,相比于原始的softmax向量量化,伪softmax向量量化的精度误差要明显小于它,最大绝对误差降低为原来的数百分之一。
然而,伪softmax与原softmax不等价,需要一些额外的操作。通常,softmax输出的注意力权重矩阵会与值矩阵进行相乘得到自注意力模块的输出,softmax向量量化是为能够使其与值矩阵计算转换为定点数矩阵运算。伪softmax向量与原softmax向量仅差归一化,即除以向量和,为线性运算。因此,可在伪softmax向量与值矩阵完成定点数运算后再进行归一,即除以伪softmax向量和,其表示如式(6)所示。
$ \begin{split}\boldsymbol{x}=\; & \frac{\boldsymbol{Q}\boldsymbol{K}^{\text{T}}}{\sqrt{d}}\text{,} \\ attention\left(\boldsymbol{Q}\text{,}\boldsymbol{K}\text{,}\boldsymbol{V}\right)=\; & softmax\left(\boldsymbol{x}\right)\cdot\boldsymbol{V}= \\ & \frac{f\left(\boldsymbol{x}\right)}{l\left(\boldsymbol{x}\right)}\cdot\boldsymbol{V}= \\ & \frac{f\left(\boldsymbol{x}\right)\cdot\boldsymbol{V}}{l\left(\boldsymbol{x}\right)}\; ,\end{split} $
其中Q,K,V分别代表自注意力模块中的查询矩阵、键矩阵和值矩阵,d代表每个注意力头的向量维度,$f(\cdot) $代表伪softmax,$l(\cdot) $则为伪softmax向量和。
对式(6)进行8b整数量化后可由式(7)表示。
$ \begin{split}attention\left({\boldsymbol{Q}}\text{,}{\boldsymbol{K}}\text{,}{\boldsymbol{V}}\right)=\frac{f\left({\boldsymbol{x}}\right)\cdot {\boldsymbol{V}}}{l\left({\boldsymbol{x}}\right)}=\frac{{s}_{{\boldsymbol{V}}}{s}_{{\rm{s}}}{{\boldsymbol{S}}}_{\text{INT8}}^{\text{w}}{{\boldsymbol{V}}}_{\text{INT}8}}{l\left({\boldsymbol{x}}\right)},\end{split} $
其中$ {s_{\boldsymbol{Q}}} $,$ {s_{\boldsymbol{K}}} $,$ {s_{\boldsymbol{V}}} $,$ {s_{\rm{s}}} $分别代表Q,K,V伪softmax输出向量的缩放系数,$ {\boldsymbol{S}}_{\mathrm{I}\mathrm{N}\mathrm{T}8}^{\mathrm{w}} $为8 b整数量化的伪softmax输出向量。通过式(7)的变换,可以将自注意力模块中的2个矩阵乘计算都转换为8 b整型计算,进一步增加了自注意力模块的量化率,提升其计算效率。
在图像分类、人脸识别等将softmax作为输出层激活函数的场景当中,其输出向量往往只关注概率最高的某一个或某几个类别,对于较小的几个峰值通常只会认为其是一种数据波动,几乎不会影响模型整体的输出准确率。对于自注意力模块而言,其通常被应用在需要对图像细节进行特征提取的场景中,如目标检测、自动驾驶感知等领域。对于自注意力模块中的softmax向量,除了需要关注某些峰值区域,同样地对于隐蔽目标的关注也需要被保留。并且,一个Transformer结构往往需要若干层自注意力模块进行串联,这就会导致softmax量化误差的指数级别的放大,从而可能对隐蔽物体完全失去注意力,降低整体模型的召回率。
例如,对于上述的多峰分布softmax向量,其中就包含较高的波峰和较小的波峰。当softmax向量的波峰信息之间的大小相差较大时,较小的波峰在量化时会更容易丢失信息,从而可能导致量化模型识别准确率降低。在下文中,称softmax输出向量的波峰为“信息峰”。
在图4中,右上图为最大信息峰的放大图,可见softmax的浮点向量和量化向量十分吻合,而左下图为最小信息峰的放大图,相比之下误差会更大。为了充分评估不同大小信息峰的量化误差,在图4中,δmean,δmax,δmin的计算由式(8)得到:
$ \begin{aligned}&{\delta }_{\text{mean}}=\dfrac{{\displaystyle \sum _{i=p}^{q}\left|\dfrac{{x}^{\text{ws}}_{{\text{f}}{\left(i\right)}}}{{s}_{\text{f}}^{\text{ws}}}-\dfrac{{x}^{\text{ws}}_{{\text{INT8}}{\left(i\right)}}}{{s}_{\text{INT8}}^{\text{ws}}}\right|}}{\left(p-q+1\right)\underset{p\le i\le q}{\mathrm{max}}{x}^{\text{ws}}_{{\text{f}}{\left(i\right)}}}\text{,}\\&{\delta }_{\text{max}}=\dfrac{\underset{p\le i\le q}{\mathrm{max}}\left|\dfrac{{x}^{\text{ws}}_{{\text{f}}{\left(i\right)}}}{{s}_{\text{f}}^{\text{ws}}}-\dfrac{{x}^{\text{ws}}_{{\text{INT8}}{\left(i\right)}}}{{s}_{\text{INT8}}^{\text{ws}}}\right|}{\underset{p\le i\le q}{\mathrm{max}}{x}^{\text{ws}}_{{\text{f}}{\left(i\right)}}}\text{,}\\&{\delta }_{\text{min}}=\dfrac{\underset{p\le i\le q}{\mathrm{min}}\left|\dfrac{{x}^{\text{ws}}_{{\text{f}}{\left(i\right)}}}{{s}_{\text{f}}^{\text{ws}}}-\dfrac{{x}^{\text{ws}}_{{\text{INT8}}{\left(i\right)}}}{{s}_{\text{INT8}}^{\text{ws}}}\right|}{\underset{p\le i\le q}{\mathrm{max}}{x}^{\text{ws}}_{{\text{f}}{\left(i\right)}}},\end{aligned} $
其中[p, q]为softmax输出向量某个信息峰的索引范围。与式(5)不同点在于,式(8)需要评估量化误差对单独1个信息峰的影响,因此需要除以信息峰的峰值以统一评估数量级。由图4所示,在最常见的多峰分布的softmax向量中,最大信息峰和最小信息峰的量化误差可以相差数十倍,即该量化方式只关注大信息峰(即峰值接近于1的信息峰)的信息精度而丢弃小信息峰(即峰值接近于0的信息峰)的信息精度。这是由于在方差较大的多峰向量中,可能会出现部分信息峰的峰值远小于1而依旧使用针对最大波峰设计的缩放系数计算,从而导致较大的误差。
通过上述分析,希望尽可能使softmax向量中的每个信息峰变大,最好是这些信息峰的峰值都能够达到或接近1,这样就能以最大绝对值1计算缩放系数来量化每个信息峰,并使其能够充分利用缩放系数的量化范围,从而降低这种缩放系数所带来的量化误差。另外,如果1个softmax向量长度足够小,其包含的信息也会更少,即信息峰会更少,那么其在量化时就可以减少对信息峰精度的影响。因此,希望将softmax向量当中的每个信息峰都单独取出来进行量化,如图5所示,即对每个信息峰单独进行伪softmax计算,这样可以使得该信息峰的峰值能够达到1。并且,希望这样的向量长度短一些,以降低量化误差对信息峰的影响。然而,信息峰的分布是黑盒的,无法判断每个softmax向量中有多少个信息峰以及这些信息峰是如何分布的。既然如此,可以考虑直接对伪softmax向量进行分块,虽然不能完全使不同的信息峰独立开来,但是能尽可能保证不同信息峰的峰值分布在不同子向量中,从而使每个子向量的最大信息峰峰值能够达到1。
为了得到分块伪softmax向量,需要将softmax分块,将其等量划分为k块,对每块进行伪softmax计算。通过分块计算,每块的伪softmax向量的最大值都能达到1,从而可以保证整个伪softmax的输出向量至少含有k个值为1且可大幅增加接近1的值,从而拉高整体的向量值以减少量化误差。如图6所示,将上述分布的向量以128维为1块,每块独立计算伪softmax,可见每一块向量块都可以达到最大值1,以使每个向量块都尽可能分布均匀。
此外,可以借鉴FlashAttention[20]利用多线程技术对自注意力模块并行计算的方式,该方法同样可以利用多线程技术对伪softmax向量进行分块量化并计算,再进行合并计算。通过此方法,伪softmax向量的量化及计算可以与前后2个矩阵乘进行合并,可以大幅降低与全局内存的交互,使得所有的计算都在寄存器上进行,减少总体的计算时间。FlashAttention采用分治算法对softmax进行分块,将对1个softmax向量的计算划分为若干子向量的计算,如式(9)所示。
$ \begin{split}m\left({\boldsymbol{x}}\right)=\;&m\left(\left({{\boldsymbol{x}}}^{1}\text{,}{{\boldsymbol{x}}}^{2}\text{,}… \text{,}{{\boldsymbol{x}}}^{k}\right)\right)=\\ &\mathrm{max}\left(m\left({{\boldsymbol{x}}}^{1}\right)\text{,}m\left({{\boldsymbol{x}}}^{2}\right)\text{,}… \text{,}m\left({{\boldsymbol{x}}}^{k}\right)\right),\end{split} $
其中x为softmax的输入向量,$ {\boldsymbol{x}}^{i}(i=1, 2,…, k) $为向量x的子向量块。$m (\cdot) $由式(2)定义。则$f ({\boldsymbol{x}}) $和$l ({\boldsymbol{x}}) $可以由式(10)计算得到:
$ \begin{aligned}f\left({\boldsymbol{x}}\right)=\;&{\left[\mathrm{exp}(m({{\boldsymbol{x}}}^{i}-m({\boldsymbol{x}}))f\left({{\boldsymbol{x}}}^{i}\right)\right]}_{i=1,2,… ,k}\text{,}\\ &l\left({\boldsymbol{x}}\right)={\displaystyle \sum _{i=1}^{k}\mathrm{exp}(m({{\boldsymbol{x}}}^{i}-m({\boldsymbol{x}}))l\left({{\boldsymbol{x}}}^{i}\right)},\end{aligned} $
其中式(10)的$f(\cdot) $和$l (\cdot) $与式(2)等价。通过该分块变换,可以由式(1)得到softmax向量:
$ \begin{split}&softmax\left( {\boldsymbol{x}} \right) = \dfrac{{f\left( {\boldsymbol{x}} \right)}}{{l\left( {\boldsymbol{x}} \right)}} =\\ &{\Biggl[{25} {\dfrac{{\exp (m({{\boldsymbol{x}}^j}) - m({\boldsymbol{x}}))f({{\boldsymbol{x}}^j})}}{{\displaystyle\sum\limits_{i = 1}^k {\exp (m({{\boldsymbol{x}}^i}) - m({\boldsymbol{x}}))f({{\boldsymbol{x}}^i})} }}} \Biggl]{25}_{j = 1,2, … ,k}}\;。\end{split} $
对于自注意力模块的8 b整数量化计算,首先需要对输入的查询矩阵、键矩阵以及值矩阵进行分块并量化,如式(12)所示。
$ \begin{split}&{\boldsymbol{Q}}={\left({{\boldsymbol{Q}}}^{1}\text{,}{{\boldsymbol{Q}}}^{2}\text{,}… \text{,}{{\boldsymbol{Q}}}^{i}\text{,}… \text{,}{{\boldsymbol{Q}}}^{{N}_{{\boldsymbol{Q}}}}\right)}^{\text{T}}\text{,}\\&{\boldsymbol{K}}={\left({{\boldsymbol{K}}}^{1}\text{,}{{\boldsymbol{K}}}^{2}\text{,}… \text{,}{{\boldsymbol{K}}}^{j}\text{,}… \text{,}{{\boldsymbol{K}}}^{{N}_{{\boldsymbol{KV}}}}\right)}^{\text{T}}\text{,}\\ &{\boldsymbol{V}}={\left({{\boldsymbol{V}}}^{1}\text{,}{{\boldsymbol{V}}}^{2}\text{,}… \text{,}{{\boldsymbol{V}}}^{j}\text{,}… \text{,}{{\boldsymbol{V}}}^{{N}_{{\boldsymbol{KV}}}}\right)}^{\text{T}}\text{,}\\&{{\boldsymbol{Q}}}_{\text{INT}8}^{i}=\left\lfloor\frac{{{\boldsymbol{Q}}}^{i}}{{s}_{{\boldsymbol{Q}}}} \right\rceil\text{,}{{\boldsymbol{K}}}_{\text{INT}8}^{j}=\left\lfloor\frac{{{\boldsymbol{K}}}^{j}}{{s}_{{\boldsymbol{K}}}} \right\rceil\text{,}{{\boldsymbol{V}}}_{\text{INT}8}^{j}=\left\lfloor \frac{{{\boldsymbol{V}}}^{j}}{{s}_{{\boldsymbol{V}}}}\right\rceil\;。\end{split} $
其中Q,K,V分别代表查询矩阵、键矩阵和值矩阵;$ {s_{\boldsymbol{Q}}} $,$ {s_{\boldsymbol{K}}} $,$ {s_{\boldsymbol{V}}} $分别为Q,K,V的缩放系数。之后便可以由式(13)分块计算伪softmax。
$ \begin{split}&{{\boldsymbol{x}}}^{ij}=\frac{{{\boldsymbol{Q}}}^{i}{{\boldsymbol{K}}}^{j}{}^{\text{T}}}{\sqrt{d}}=\frac{{s}_{{\boldsymbol{Q}}}{s}_{{\boldsymbol{K}}}}{\sqrt{d}}{{\boldsymbol{Q}}}_{\text{INT}8}^{i}{{\boldsymbol{K}}}_{\text{INT}8}^{j}{}^{\text{T}}\text{,}\\&{\hat{{\boldsymbol{S}}}}^{ij}=\mathrm{exp}({{\boldsymbol{x}}}^{ij}-\underset{i}{\mathrm{max}}({{\boldsymbol{x}}}^{ij})),\end{split} $
其中${{\boldsymbol{\hat S}}^{ij}}$为分块伪softmax输出矩阵块, $\mathop {\max }\limits_i ({{\boldsymbol{x}}^{ij}})$为在第i块上对j维度上的向量求最大值。之后便可以对分块伪softmax进行量化,并与值矩阵块进行定点数计算,如式(14)所示。
$ \begin{split}&{s}_{{\rm{s}}}=\frac{1}{255}\text{,}\text{ }{\hat{{\boldsymbol{S}}}}_{\text{INT}8}^{ij}=\left\lfloor\frac{{\hat{{\boldsymbol{S}}}}^{ij}}{{s}_{{\rm{s}}}} \right\rceil\text{,}\\&{\hat{{\boldsymbol{S}}}}^{ij}{{\boldsymbol{V}}}^{j}={s}_{{\rm{s}}}{s}_{{\boldsymbol{V}}}\cdot {\hat{{\boldsymbol{S}}}}_{\text{INT}8}^{ij}{{\boldsymbol{V}}}_{\text{INT}8}^{j}\;。\end{split} $
之后便可以根据式(9)和式(10)进行合并,得到:
$ \begin{split}{{\boldsymbol{x}}}_{\text{m}}^{ij}=\;&\underset{i}{\mathrm{max}}\left({{\boldsymbol{x}}}^{ij}\right)\text{,}{m}_{i}=\underset{j=1,2,… ,{N}_{{\boldsymbol{KV}}}}{\mathrm{max}}\left({{\boldsymbol{x}}}_{\text{m}}^{ij}\right)\text{,}\\{\hat{{\boldsymbol{S}}}}^{i}{\boldsymbol{V}}=\;&{\displaystyle \sum _{j}\mathrm{exp}}({{\boldsymbol{x}}}_{\text{m}}^{ij}-{m}_{i}){\hat{{\boldsymbol{S}}}}^{ij}{{\boldsymbol{V}}}^{j}=\\&{\displaystyle \sum _{j}\mathrm{exp}}\left(\underset{i}{\mathrm{max}}\left(\frac{{{\boldsymbol{Q}}}^{i}{{\boldsymbol{K}}}^{j}{}^{\text{T}}}{\sqrt{d}}\right)-{m}_{i}\right){\hat{{\boldsymbol{S}}}}^{ij}{{\boldsymbol{V}}}^{j}\;。\end{split} $
因为伪softmax未进行归一化操作,需要对得到的输出除以伪softmax的和以与原来的自注意力模块计算等价,如式(16)所示。
$ \begin{split}{{\boldsymbol{S}}}^{i}=\;&\frac{{\hat{{\boldsymbol{S}}}}^{i}}{l\left({{\boldsymbol{x}}}^{i}\right)}\text{,}l\left({{\boldsymbol{x}}}^{ij}\right)={\displaystyle \sum {{\boldsymbol{x}}}^{ij}}\text{,}\\ l\left({{\boldsymbol{x}}}^{i}\right)=\;&{\displaystyle \sum _{j}\left(\mathrm{exp}\left(\underset{i}{\mathrm{max}}\left(\frac{{{\boldsymbol{Q}}}^{i}{{\boldsymbol{K}}}^{j}{}^{\text{T}}}{\sqrt{d}}\right)-{m}_{i}\right)l\left({{\boldsymbol{x}}}^{ij}\right)\right)}\text{,}\\&attention\left({{\boldsymbol{Q}}}^{i}\text{,}{\boldsymbol{K}}\text{,}{\boldsymbol{V}}\right)={{\boldsymbol{S}}}^{i}{\boldsymbol{V}}=\frac{{\hat{{\boldsymbol{S}}}}^{i}{\boldsymbol{V}}}{l\left({{\boldsymbol{x}}}^{i}\right)} =\\&\frac{{\displaystyle \sum _{j}\left(\mathrm{exp}\left(\underset{i}{\mathrm{max}}\left(\frac{{{\boldsymbol{Q}}}^{i}{{\boldsymbol{K}}}^{{j}{\text{T}}}}{\sqrt{d}}\right)-{m}_{i}\right){\hat{{\boldsymbol{S}}}}^{ij}{{\boldsymbol{V}}}^{j}\right)}}{{\displaystyle \sum _{j}\left(\mathrm{exp}\left(\underset{i}{\mathrm{max}}\left(\frac{{{\boldsymbol{Q}}}^{i}{{\boldsymbol{K}}}^{j}{}^{\text{T}}}{\sqrt{d}}\right)-{m}_{i}\right)l\left({{\boldsymbol{x}}}^{ij}\right)\right)}},\end{split} $
其中mi为j维度向量的全局最大值。考虑到查询矩阵块$ {\boldsymbol{Q}}^{i} $之间的计算是独立的,因此第i个查询矩阵块经过自注意力模块计算便可以当作输出的第i块。
同样地,采用均匀、单峰、双峰以及多峰分布的输入进行误差分析。与图3相比,图7在同个区域的整型量化明显有更多的划分层,并与浮点向量更贴合。同样地,为了与之前的误差评估方式一致,需要对每一个向量块转换回去后再计算误差,$ {\delta }_{\mathrm{m}\mathrm{e}\mathrm{a}\mathrm{n}} $,$ {\delta }_{\mathrm{m}\mathrm{a}\mathrm{x}} $,$ {\delta }_{\mathrm{m}\mathrm{i}\mathrm{n}} $的计算由式(17)得到。
$ \begin{aligned}&{\delta _{{\text{mean}}}} = \dfrac{{\displaystyle\sum\limits_j {\displaystyle\sum\limits_{i = {p_j}}^{{q_j}} {\left| {\dfrac{{x^{{\text{kws}}{\left( j \right)}}_{{\text{f}}{\left( i \right)}}}}{{s_{\text{f}}^{{\text{ws}}}}} - \dfrac{{x_{{\text{INT8}}\left( i \right)}^{{\text{kws}}\left( j \right)}}}{{s_{{\text{INT8}}}^{{\text{ws}}}}}} \right|} \mathop {\max }\limits_i x_{{\text{f}}\left( i \right)}^{{\text{ws}}\left( j \right)}} }}{n}, \\ &{\delta _{{\text{max}}}} = \mathop {\max }\limits_j \mathop {\max }\limits_{{p_j} \leqslant i \leqslant {q_j}} \left| {\dfrac{{x_{{\text{f}}\left( i \right)}^{{\text{kws}}\left( j \right)}}}{{s_{\text{f}}^{{\text{ws}}}}} - \dfrac{{x_{{\text{INT8}}\left( i \right)}^{{\text{kws}}\left( j \right)}}}{{s_{{\text{INT8}}}^{{\text{ws}}}}}} \right|\mathop {\max }\limits_i x_{{\text{f}}\left( i \right)}^{{\text{ws}}\left( j \right)}, \\ &{\delta _{{\text{min}}}} = \mathop {\min }\limits_j \mathop {\min }\limits_{{p_j} \leqslant i \leqslant {q_j}} \left| {\dfrac{{x_{{\text{f}}\left( i \right)}^{{\text{kws}}\left( j \right)}}}{{s_{\text{f}}^{{\text{ws}}}}} - \dfrac{{x_{{\text{INT8}}\left( i \right)}^{{\text{kws}}\left( j \right)}}}{{s_{{\text{INT8}}}^{{\text{ws}}}}}} \right|\mathop {\max }\limits_i x_{{\text{f}}\left( i \right)}^{{\text{ws}}\left( j \right)},\end{aligned} $
其中$ {{x}_{\mathrm{f}\left(i\right)}^{\mathrm{k}\mathrm{w}\mathrm{s}\left(j\right)}} $和$ {{x}_{\mathrm{I}\mathrm{N}\mathrm{T}8\left(i\right)}^{\mathrm{k}\mathrm{w}\mathrm{s}\left(j\right)}} $分别为第j块伪softmax浮点向量块和量化向量块的第i个值,而$ {{x}_{\mathrm{f}\left(i\right)}^{\mathrm{w}\mathrm{s}\left(j\right)}} $为原始伪softmax浮点向量中与第j块伪softmax浮点向量块对应的子向量的第i个值,[$ {p}_{j} $, $ {q}_{j} $]表示第j块伪softmax向量块的索引范围。根据分块伪softmax的定义,伪softmax浮点向量块$ {{\boldsymbol{x}}_{\mathrm{f}}^{\mathrm{k}\mathrm{w}\mathrm{s}\left(j\right)}} $和原始伪softmax浮点向量的子向量$ {{\boldsymbol{x}}_{\mathrm{f}}^{\mathrm{w}\mathrm{s}\left(j\right)}} $具有如式(18)所示的关系:
$ {\boldsymbol{x}}_{\text{f}}^{{\text{kws}}\left( j \right)} = \frac{{{\boldsymbol{x}}_{\text{f}}^{{\text{ws}}\left( j \right)}}}{{\mathop{\max }\limits_i {\boldsymbol{x}}_{{\text{f}}\left( i \right)}^{{\text{ws}}\left( j \right)}}}\;。$
由图7可见,通过分块伪softmax量化方式,伪softmax向量块的量化误差有一定的下降。虽然整体向量的量化误差没有数量级级别的下降,但是分块的方式可以使每个小信息峰转变为大信息峰,从而降低小信息峰的量化误差。对于自注意力模块而言,这些小信息峰的信息也十分重要,在后面对算子的实验中论证了这一点。
为了更显著地表示分块伪softmax方法对多峰向量中不同大小信息峰量化误差的影响,图8展示了与图4中相同信息峰在分块伪softmax方法下的量化误差图。可见,通过分块伪softmax量化的方法,小信息峰的精度可以达到甚至超过大信息峰的精度。相比于全局伪softmax量化方法,分块伪softmax量化方法可以使多峰向量中的小信息峰的量化误差下降为原来的数百分之一,充分保留了softmax输出向量的全部精度信息,不会像全局伪softmax量化方法那样侧重于大信息峰的量化精度。
自注意力模块通过对查询矩阵和键矩阵的乘积进行softmax计算得到权重系数矩阵,再与值矩阵进行加权计算得到自注意力输出值。其输入通常为查询矩阵、键矩阵以及值矩阵,其中查询矩阵的大小为 ($ {L}_{\boldsymbol{Q}} $, B, H×d),而键矩阵与值矩阵的大小均为 ($ {L}_{\boldsymbol{K}\boldsymbol{V}} $, B, H×d),其中$ {L}_{\boldsymbol{Q}} $和$ {L}_{\boldsymbol{K}\boldsymbol{V}} $分别表示查询矩阵以及键或值矩阵的序列长度,B表示批数据大小,H表示注意力头的个数,d表示每个注意力头的通道数。
如3.2节所述,为了尽可能降低8 b整数自注意力模块的精度损失,需要对softmax分块量化和计算,并可以通过FlashAttention的方式与前后2个矩阵乘法进行合并,以降低显存的读写量。
8 b整数的自注意力模块算子实现流程为:
1)对查询矩阵进行分块,每块64×d,并在每个线程中对键矩阵和值矩阵分块遍历,每块同样为64×d,且每个线程块的线程数为128。
2)每个线程束(32线程)负责1个查询矩阵块的子块16×d的计算,及16×d的查询矩阵块子块与键矩阵块64×d的矩阵乘,利用MMA(matrix multiply-accumulate)指令集在Tensor Core上计算。在图灵架构的显卡下,1个Tensor Core最多可以支持8×16和16×8的矩阵乘;而在安培架构的显卡下,最多可以支持16×32和32×8的矩阵乘。查询矩阵块与键矩阵块的矩阵乘计算如式(19)所示。
$ \begin{split}&{{\boldsymbol{Q}}}_{\text{INT}8}^{i}{K}_{\text{INT}8}^{j}{}^{\text{T}}=\\&\left(\begin{array}{ccc}{{\boldsymbol{q}}}_{1,1}& … & {{\boldsymbol{q}}}_{1,\tfrac{d}{k}}\\\vdots & & \vdots \\{{\boldsymbol{q}}}_{\tfrac{16}{m},1}& … & {{\boldsymbol{q}}}_{\tfrac{16}{m},\tfrac{d}{k}}\end{array}\right){\left(\begin{array}{ccc}{{\boldsymbol{k}}}_{1,1}& … & {{\boldsymbol{k}}}_{1,\tfrac{d}{k}}\\\vdots & & \vdots \\ {{\boldsymbol{k}}}_{\tfrac{64}{n},1}& … & {{\boldsymbol{k}}}_{\tfrac{64}{n},\tfrac{d}{k}}\end{array}\right)}^{\text{T}},\end{split}$
其中$ {\boldsymbol{Q}}_{\mathrm{I}\mathrm{N}\mathrm{T}8}^{i} $和$ {\boldsymbol{K}}_{\mathrm{I}\mathrm{N}\mathrm{T}8}^{j} $分别为8 b整数的查询矩阵块和键矩阵块,d为每个自注意力头的通道维度,$ {\boldsymbol{q}}_{i,j} $和$ {\boldsymbol{k}}_{i,j} $分别为查询矩阵块和键矩阵块的子矩阵,m,k,n分别为1个Tensor Core计算单元能计算的矩阵维度,在安培架构中m=16,k=32,n=8。
3)将根据式(12)得到的矩阵块反量化为浮点数后,在最后1个维度上计算伪softmax块。该伪softmax块的值范围为[0,1],且每行都至少存在1个最大值“1”,分布充分均匀。
4)将得到的浮点数伪softmax块根据式(13)进行量化,再与值矩阵块相乘。矩阵块的相乘同样使用MMA指令集在Tensor Core上计算,如式(20)所示。
$ \begin{split}&{\hat{{\boldsymbol{S}}}}_{\text{INT}8}^{ij}{{\boldsymbol{V}}}_{\text{INT}8}^{j}=\\&\left(\begin{array}{ccc}{\hat{{\boldsymbol{s}}}}_{1,1}& … & {\hat{{\boldsymbol{s}}}}_{1,\tfrac{64}{k}}\\\vdots & & \vdots \\{\hat{{\boldsymbol{s}}}}_{\tfrac{16}{m},1}& … & {\hat{{\boldsymbol{s}}}}_{\tfrac{16}{m},\tfrac{64}{k}}\end{array}\right)\left(\begin{array}{ccc}{{\boldsymbol{v}}}_{1,1}& … & {{\boldsymbol{v}}}_{1,\tfrac{d}{k}}\\\vdots & & \vdots \\ {{\boldsymbol{v}}}_{\tfrac{64}{n},1}& … & {{\boldsymbol{v}}}_{\tfrac{64}{n},\tfrac{d}{k}}\end{array}\right),\end{split}$
其中${\boldsymbol{\hat S}}_{{\text{INT}}8}^{ij}$和${\boldsymbol{V}}_{{\text{INT}}8}^j$分别为8 b整数的伪softmax输出块与值矩阵块,${{\boldsymbol{\hat s}}_{i,j}}$,${{\boldsymbol{v}}_{i,j}}$分别为伪softmax输出块和键矩阵块的子矩阵块。
5)根据式(15)和式(16)将各个值矩阵块根据伪softmax块加权相乘后合并,合并完后再除以伪softmax的每行和,以对其进行归一化。
由于分块伪softmax量化需要与前后2个矩阵乘法进行合并,即每个线程都会承担矩阵块乘法即伪softmax块的量化与反量化。通常,伏特架构的英伟达显卡开始支持了更加高效的Tensor Core编程,其在硬件层面上对小矩阵乘法做了深度优化,使用1个线程簇(32线程)计算矩阵块乘法,显著提升了矩阵的计算速度。
调用Tensor Core的方式主要有2种:WMMA(warp level matrix multiply-accumulate)指令集与MMA指令集。WMMA是一种高级指令集,可以非常方便调用Tensor Core模块,但是需要共享内存作为矩阵存储媒介。而MMA是一种低级指令集,与硬件版本相关,需要了解1个矩阵块在32个线程中的输入输出存储分布,可直接在寄存器上进行操作,无需共享内存介入。
考虑到8 b定点数算子的实现需要进行反复的量化与反量化,使用更加底层的MMA指令集进行矩阵乘法,便可以在寄存器上操作数据的量化与反量化,避免频繁的量化与反量化操作给8 b定点数算子带来额外的数据读写延时。
在第3节中,讨论并分析了关于softmax的多种量化方式以及精度损失。下面针对自注意力模块算子对这几种量化方式进行消融实验。本节实验对比了自注意力模块的英伟达官方闭源版本(含单精度与半精度对比)、softmax直接量化版本、伪softmax量化版本以及分块伪softmax量化版本(含WMMA指令集与MMA指令集对比)。
表1展示了自注意力模块的浮点计算与8 b整数量化的实验结果。可以看出,实验复现的自注意力模块版本在性能上接近官方闭源版本,这为后续的量化实验提供了可靠的基准。
在量化实验方面,尝试了直接量化、伪向量量化以及分块伪向量量化3种不同的方法。从算子平均绝对误差的角度来看,这3种方法的误差是逐级减少的,这说明随着量化策略的优化,所提方法能够逐渐减少量化带来的精度损失。
此外,实验还利用了MMA底层指令集的支持,将softmax计算前后的量化与反量化数据都在寄存器中执行,避免了共享内存的介入。这种优化策略不仅提高了数据处理的效率,还大幅降低了在GPU上的计算耗时。这对于实际应用中提高模型的推理速度和降低资源依赖具有重要意义。
通过表1还可以发现,全局softmax量化的耗时会比半精度浮点数计算要高。对此,本节设计一些实验,对这部分的异常实验结果进行分析。
对自注意力模块中3个部分即2个矩阵乘以及1个softmax算子分别测速分析,实验结果如表2所示。
由表2可以得到,8 b整数量化的2个矩阵乘法的速度不降反升,这是不符合量化计算特性的。更进一步观察可以发现,对于矩阵乘(前),其输入矩阵大小分别为(256, 1 024, 32)和(256, 32, 1024),而输出矩阵大小为(256, 1024, 1024)。可见,输入和输出矩阵大小相差巨大,这会导致写入数据会远大于读出输出。从而可以计算出半精度和8 b整数量化下的矩阵乘(前)的读写量。
表3展示了矩阵乘(前)模块的数据读写量。由于8 b量化计算的输出是32 b整数(4 B),相比于半精度计算输出仍为半精度(2 B),其数据写入量要高1倍。而查询矩阵与键矩阵的矩阵乘计算输出矩阵要远大于输入矩阵,因此在该矩阵乘中,其数据写入量会决定其耗时。同样地,对于矩阵乘(后)的计算,其也对GPU的全局内存进行了大量的写入,因此无法发挥8 b整数量化的计算效率。
而对于分块量化方法而言,因为其对softmax进行了分块计算,从而可以与前后2个矩阵乘法进行分块合并计算,无需对GPU的全局内存进行额外的读写,可以大幅提升其计算速度。因此,采用分块伪softmax向量量化方法,不仅可以在一定程度上降低量化损失,还能大幅提升计算速度。
本节实验以模型单精度浮点数推理作为精度对照组,分别采用伪softmax向量量化方法以及分块伪softmax向量量化方法对自注意力机制的softmax向量进行8 b整数量化,并部署到主流的多目3D目标检测模型BEVFormer[21]中计算检测精度。
实验对BEVFormer的3种规格大小(tiny,small,base)的模型均进行实验,评测指标为自动驾驶开源3D检测数据集NuScenes[22]在多目3D目标检测任务上的评测指标NDS(NuScenes detection score),实验结果如表4所示。
对自注意力机制中的softmax向量执行伪softmax向量量化后,3种规格大小的BEVFormer模型的NDS均出现了2~3个百分点的下降,而采用分块伪softmax向量量化方法的3种规格大小的BEVFormer模型相比于单精度浮点数推理精度几乎没有精度损失。
该实验充分论证了分块伪softmax向量量化方法对模型精度保留的有效性,其可以在不影响模型整体精度的情况下,提升整体模型8 b量化计算的占比,从而提升模型整体的计算速度并降低对资源的依赖程度。
从实验结果来看,相比于原始的softmax向量直接量化方法,所提出的伪softmax向量量化和分块伪softmax向量量化方法的算子平均绝对误差均有显著的下降。其次,考虑到自注意力模块在8 b量化计算模式下对全局内存的读写量相比于半精度计算存在较大的不平衡现象,直接导致了伪softmax向量量化的计算效率下降。对于分块伪softmax向量量化方法,本文使用了英伟达图形处理器中底层的矩阵计算指令(WMMA和MMA)对该算子的矩阵计算进行了合并,避免了中间计算结果对全局内存的频繁读写操作,从而大幅提升了分块伪softmax向量量化算子的计算效率。
需要注意的是,对于分块伪softmax向量量化计算,其并不完全适用于所有自注意力模块场景。考虑到其是采用底层指令进行计算,底层指令对矩阵大小通常会有一定要求。在本文方法实现中,要求自注意力模块的输入向量序列长度需要为128的整数倍,以便于底层计算指令的高效计算。对于大部分深度学习模型而言,其内部参数的长度通常满足要求。因此,所提出的分块伪softmax向量量化可以适用于目前大部分的深度学习模型。
本文针对自注意力模块直接量化方法的量化误差进行了深入的分析,并指出了误差较大的原因。随后,提出了伪softmax向量量化和分块伪softmax量化方法,分析并解决了softmax量化精度的问题。通过对浮点计算的算子和各个整数量化方法计算的算子进行了充分的实验,评估其输出的平均绝对误差和算子耗时,论证了所提出方法的有效性。
作者贡献声明:林德铝提出算法思路,设计和执行实验,撰写论文;何琨提出指导意见并修改论文。

参考文献 引证文献
排序方式:
1
Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C]//Proc of the 31st Conf on Neural Information Processing Systems. Red Hook, NY: Curran Associates Inc, 2017: 5998−6008
2
廖兴滨, 秦小林, 张思齐, 等. 交互式机器翻译综述[J]. 计算机应用, 2023, 43(2): 329−334
Liao Xingbin, Qin Xiaolin, Zhang Siqi, et al. Review of interactive machine translation[J]. Journal of Computer Applications, 2023, 43(2): 329−334 (in Chinese)
3
刘迪, 奚雪峰, 崔志明, 等. 抽取-生成式自动文本摘要技术研究综述[J]. 计算机技术与发展, 2023, 33(5): 1−8
Liu Di, Xi Xuefeng, Cui Zhiming, et al. Review of research on extractive-abstractive automatic text summarization technology[J]. Computer Technology and Development, 2023, 33(5): 1−8 (in Chinese)
4
陈健鹏, 马建辉, 王怡君. 基于多轮交互的人机对话系统综述[J]. 南京信息工程大学学报, 2019, 11(3): 256−268
Chen Jianpeng, Ma Jianhui, Wang Yijun. A survey of human-computer dialogue system based on multiple-round interaction[J]. Journal of Nanjing University of Information Science & Technology, 2019, 11(3): 256−268 (in Chinese)
5
Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16x16 words: Transformers for image recognition at scale[C/OL]//Proc of the 9th Int Conf on Learning Representations. Washington: ICLR, 2021[2025-03-29]. https://openreview.net/forum?id=YicbFdNTTy
6
Deng J, Dong W, Socher R, et al. ImageNet: A large-scale hierarchical image database[C]//Proc of the 22nd IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2009: 248−255
7
Radford A, Narasimhan K, Salimans T, et al. Improving language understanding with unsupervised learning[R]. San Francisco: OpenAI, 2018
8
Floridi L, Chiriatti M. GPT-3: Its nature, scope, limits, and consequences[J]. Minds and Machines, 2020, 30(4): 681−694
9
Touvron H, Lavril T, Izacard G, et al. LLaMA: Open and efficient foundation language models[J]. arXiv preprint, arXiv: 2302.13971, 2023
10
Ioffe S, Szegedy C. Batch normalization: Accelerating deep network training by reducing internal covariate shift[C]//Proc of the 32nd Int Conf on Machine Learning. Stroudsburg, PA: ACL, 2015: 448−456
11
Jacob B, Kligys S, Chen B, et al. Quantization and training of neural networks for efficient integer-arithmetic-only inference[C]//Proc of the 31st IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2018: 2704−2713
12
Sun Xiao, Choi J, Chen C Y, et al. Hybrid 8-bit floating point training and inference for deep neural networks[C]//Proc of the 33rd Conf on Neural Information Processing Systems. Red Hook, NY: Curran Associates Inc, 2019: 4901−4910
13
Lin Ji, Tang Jiaming, Tang Haotian, et al. AWQ: Activation-aware weight quantization for LLM compression and acceleration[C/OL]//Proc of the 7th Conf on Machine Learning and Systems. 2024[2025-03-29]. https://proceedings.mlsys.org/paper_files/paper/2024/file/42a452cbafa9dd64e9ba4aa95cc1ef21-Paper-Conference.pdf
14
Dettmers T, Pagnoni A, Holtzman A, et al. Qlora: Efficient finetuning of quantized LLMs[C]// Proc of the 37th Conf on Neural Information Processing Systems. Red Hook, NY: Curran Associates Inc, 2023: 10088−10115
15
Lin Yang, Zhang Tianyu, Sun Peiqin, et al. FQ-ViT: Post-training quantization for fully quantized vision transformer[C]//Proc of the 31st Int Joint Conf on Artificial Intelligence. Burlington: Morgan Kaufmann, 2022: 1173−1179
16
Yuan Zhihang, Xue Chenhao, Chen Yiqi, et al. PTQ4ViT: Post-training quantization for vision transformers with twin uniform quantization[C]//Proc of the 19th European Conf on Computer Vision. Berlin: Springer, 2022: 191−207
17
Wang Meiqi, Lu Siyuan, Zhu Danyang, et al. A high-speed and low-complexity architecture for softmax function in deep learning[C]//Proc of the 14th IEEE Asia Pacific Conf on Circuits and Systems. Piscataway, NJ: IEEE, 2018: 223−226
18
Vasyltsov I, Chang W. Efficient softmax approximation for deep neural networks with attention mechanism[J]. arXiv preprint, arXiv: 2111.10770, 2021
19
张强, 杨吉斌, 张雄伟, 等. CS-Softmax: 一种基于余弦相似性的Softmax损失函数[J]. 计算机研究与发展, 2022, 59(4): 936−949
Zhang Qiang, Yang Jibin, Zhang Xiongwei, et al. CS-Softmax: A cosine similarity-based Softmax loss function[J]. Journal of Computer Research and Development, 2022, 59(4): 936−949 (in Chinese)
20
Dao T, Fu D Y, Ermon S, et al. FlashAttention: Fast and memory-efficient exact attention with IO-awareness[C]//Proc of the 36th Conf on Neural Information Processing Systems. Red Hook, NY: Curran Associates Inc, 2022: 16344−16359
21
Li Zhiqi, Wang Wenhai, Li Hongyang, et al. BEVFormer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers[C]//Proc of the 19th European Conf on Computer Vision. Berlin: Springer, 2022: 1−18
22
Caesar H, Bankiti V, Lang A H, et al. NuScenes: A multimodal dataset for autonomous driving[C]//Proc of the 33rd IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2020: 11618−11628
2026年第63卷第1期
178
74
0
引用本文
BibTeX
文章信息
doi: 10.7544/issn1000-1239.202440804
  • 接收时间:2024-10-19
  • 首发时间:2026-11-06
  • 出版时间:2026-01-01
补充材料
相关文章
文章信息
作者
出版历史
  • 收稿日期:2024-10-19
  • 修回日期:2025-05-19
  • 录用日期:2025-06-11
基金
作者信息
    华中科技大学计算机科学与技术学院 武汉 430074

通讯作者:

参考文献
分享链接
https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202440804
分享至
全文二维码

扫描看全文

引用本文
BibTeX
本文的引用情况
2种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏