Article(id=1325490262708977701, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1325489890678420381, articleNumber=null, orderNo=null, doi=10.7544/issn1000-1239.202550040, pmid=null, cstr=null, oa=null, hot=null, price=null, onlineType=0, articleFormat=0, articleType=null, articleTypeStr=research-article, receivedDate=1736870400000, receivedDateStr=2025-01-15, revisedDate=1756224000000, revisedDateStr=2025-08-27, acceptedDate=1756742400000, acceptedDateStr=2025-09-02, onlineDate=1793951105047, onlineDateStr=2026-11-06, pubDate=1767196800000, pubDateStr=2026-01-01, doiRegisterDate=null, doiRegisterDateStr=null, onlineIssueDate=1793951105047, onlineIssueDateStr=2026-11-06, onlineJustAcceptDate=null, onlineJustAcceptDateStr=null, onlineFirstDate=null, onlineFirstDateStr=null, sourceXml=null, magXml=null, createTime=1793951105047, creator=13701087609, updateTime=1793951105047, updator=13701087609, issue=Issue{id=1325489890678420381, tenantId=1146029695717560320, journalId=1324759914146529478, year='2026', volume='63', issue='1', pageStart='1', pageEnd='272', issueExtLink='null', onlineDate='null', pubDate='1767196800000', pubDateStr='2026-01-01', beforeIssueId=null, nextIssueId=null, price=null, status=1, issueComplete=1, articleOrder=1, issueType=-1, specialIssue=null, createTime=1793951016348, creator='13701087609', updateTime=1793952522017, updator='13701087609', preIssue=null, nextIssue=null, articleTotal=null, ext={EN=IssueExt(id=1325496205974839692, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1325489890678420381, language=EN, specialIssueTitle=, coverIllustrator=null, specialIssueEditor=, specialIssueAbout=), CN=IssueExt(id=1325496205974839693, tenantId=1146029695717560320, journalId=1324759914146529478, issueId=1325489890678420381, language=CN, specialIssueTitle=, coverIllustrator=null, specialIssueEditor=, specialIssueAbout=)}, issueFiles=null, downloadFileDto=null}, startPage=147, endPage=161, ext={EN=ArticleExt(id=1325490268144795686, articleId=1325490262708977701, tenantId=1146029695717560320, journalId=1324759914146529478, language=EN, title=Dual-Perspective Multi-Level Cross-Modal Recommendation Based on Large Language Models, columnId=1325489891789910943, journalTitle=Journal of Computer Research and Development, columnName=Artificial Intelligence, runingTitle=null, highlight=null, articleAbstract=

Multimodal recommendation systems aim to deliver more accurate and personalized recommendations by leveraging diverse data modalities such as text and images. Despite their potential, existing approaches face several critical challenges. 1) Feature distortion. The input embeddings are processed by relatively small pre-trained language models and deep convolutional neural networks, which may lead to inaccurate and incomplete feature representations. 2) Single encoding perspective. The multimodal encoding layers of current models only consider encoding from a single perspective that is either memory-based or expansion-based, which limits representational capacity and leads to information loss. 3) Poor multimodal alignment. Embeddings from different modalities are distributed in distinct and heterogeneous semantic spaces, and need to be effectively mapped into a shared semantic space to achieve accurate alignment. However, existing methods typically rely on simple operations such as direct product of behavioral information, which cannot adequately capture the complex and high-order relationships among modalities. As a result, precise alignment across multiple modalities is difficult to achieve. To address these issues, we propose a novel model named DPRec. This model considers encoding from both memory and expansion perspectives and introduces hypergraphs for multi-level precise cross-modal alignment. The model is evaluated through extensive experiments on three real-world datasets. Experimental results confirm the effectiveness of DPRec, demonstrating its superiority over state-of-the-art approaches in improving recommendation accuracy.

, authors=Yiming Li, Yaxin Yu*, Zhisheng Yu, Yiting Si, Yusong Ye, authorsList=Yiming Li, Yaxin Yu, Zhisheng Yu, Yiting Si, Yusong Ye, authorCompany=null, correspAuthors=Yaxin Yu, authorNote=

born in 2000. Master candidate. His main research interests include knowledge graph and recommendation system

born in 1971. PhD, associate professor, master supervisor. Member of IEEE, ACM and CCF. Her main research interests include data mining and social network

born in 2001. Master candidate. His main research interests include recommendation system and causal relationship inference

born in 2001. Master candidate. His main research interests include recommendation system and large language model

born in 2001. Master candidate. His main research interests include data mining and recommendation system

, correspAuthorsNote=null, copyrightStatement=Copyright © 2026 Journal of Computer Research and Development. All rights reserved., copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=null, magXml=null, pdfUrl=null, pdf=null, pdfFileSize=null, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=null, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=null, mapNumber=null, fund=null), CN=ArticleExt(id=1325490350365737034, articleId=1325490262708977701, tenantId=1146029695717560320, journalId=1324759914146529478, language=CN, title=基于大语言模型的双视角多级跨模态推荐, columnId=1325489892104483745, journalTitle=计算机研究与发展, columnName=人工智能, runingTitle=null, highlight=null, articleAbstract=

多模态推荐系统旨在提供更为精准和个性化的推荐服务。然而,现有研究仍存在以下问题:1)特征失真。由于输入的嵌入均由小型预训练语言模型和深层卷积神经网络等模型进行处理,导致得到的特征表示不准确。2)编码视角单一。目前模型的多模态编码层只考虑在单一的记忆或扩展视角进行编码,造成信息缺失。3)多模态对齐效果欠佳。不同模态嵌入分布在不同空间中,需将其映射至同一空间以实现对齐。而现有方法通过简单的行为信息乘积无法捕捉模态之间的复杂关系,导致多种模态无法精确对齐。基于上述问题,提出了一个新颖的模型DPRec。该模型同时考虑了记忆与扩展的双视角编码,并引入超图进行多级精准跨模态对齐。所提模型在3个真实数据集上进行了扩展实验,实验结果验证了所提模型的有效性。

, authors=李佚名, 于亚新*, 于之晟, 司一廷, 叶育松, authorsList=李佚名, 于亚新, 于之晟, 司一廷, 叶育松, authorCompany=null, correspAuthors=于亚新, authorNote=

2000年生。硕士研究生。主要研究方向为知识图谱、推荐系统

1971年生。博士,副教授,硕士生导师。IEEE,ACM,CCF会员。主要研究方向为数据挖掘、社交网络

2001年生。硕士研究生。主要研究方向为推荐系统、因果关系推理

2001年生。硕士研究生。主要研究方向为推荐系统、大语言模型

2001年生。硕士研究生。主要研究方向为数据挖掘、推荐系统

, correspAuthorsNote=
于亚新()
, copyrightStatement=版权所有 © 2026 计算机研究与发展编辑部, copyrightOwner=null, extLink=null, articleAbsUrl=null, sourceXml=BIFGb9yGKpspSSXdyluvrg==, magXml=VcKYv+PunqtwjA3gJ29+JQ==, pdfUrl=null, pdf=fuI4plHIGDtEkmnQa2g78g==, pdfFileSize=2643382, pdfExtLink=null, richHtmlUrl=null, mobilePdfUrl=null, reviewReport=null, pdfFirstPage=null, abstractGraph=KiLe02Huc1uETDAg520Sig==, abstractGraphContent=null, abstractVideo=null, citation=null, cebUrl=null, magXmlContent=5sTAZIYrPuGargxbjuLqvA==, mapNumber=null, fund=null)}, authors=[Author(id=1325490359236690004, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=2372016@stu.neu.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325490359647731797, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, authorId=1325490359236690004, language=EN, stringName=Yiming Li, firstName=Yiming, middleName=null, lastName=Li, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=null, bio={"content":"

born in 2000. Master candidate. His main research interests include knowledge graph and recommendation system

"}, bioImg=null, bioContent=

born in 2000. Master candidate. His main research interests include knowledge graph and recommendation system

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325490359979081814, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, authorId=1325490359236690004, language=CN, stringName=李佚名, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=null, bio={"img":"TZl6+eeluZ2NnrbxcqrDnA==","content":"

2000年生。硕士研究生。主要研究方向为知识图谱、推荐系统

"}, bioImg=TZl6+eeluZ2NnrbxcqrDnA==, bioContent=

2000年生。硕士研究生。主要研究方向为知识图谱、推荐系统

, aboutCorrespAuthor=null)}, companyList=null), Author(id=1325490360482398296, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=yuyx@mail.neu.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=1, authorType=1, ext={EN=AuthorExt(id=1325490363779121241, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, authorId=1325490360482398296, language=EN, stringName=Yaxin Yu, firstName=Yaxin, middleName=null, lastName=Yu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=*, address=null, bio={"content":"

born in 1971. PhD, associate professor, master supervisor. Member of IEEE, ACM and CCF. Her main research interests include data mining and social network

"}, bioImg=null, bioContent=

born in 1971. PhD, associate professor, master supervisor. Member of IEEE, ACM and CCF. Her main research interests include data mining and social network

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325490365930799194, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, authorId=1325490360482398296, language=CN, stringName=于亚新, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=*, address=null, bio={"img":"5bQINdYT1R3SJn2G1c3uew==","content":"

1971年生。博士,副教授,硕士生导师。IEEE,ACM,CCF会员。主要研究方向为数据挖掘、社交网络

"}, bioImg=5bQINdYT1R3SJn2G1c3uew==, bioContent=

1971年生。博士,副教授,硕士生导师。IEEE,ACM,CCF会员。主要研究方向为数据挖掘、社交网络

, aboutCorrespAuthor=null)}, companyList=null), Author(id=1325490366782242908, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325490367973425245, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, authorId=1325490366782242908, language=EN, stringName=Zhisheng Yu, firstName=Zhisheng, middleName=null, lastName=Yu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=null, bio={"content":"

born in 2001. Master candidate. His main research interests include recommendation system and causal relationship inference

"}, bioImg=null, bioContent=

born in 2001. Master candidate. His main research interests include recommendation system and causal relationship inference

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325490368501907550, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, authorId=1325490366782242908, language=CN, stringName=于之晟, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=null, bio={"img":"NoqKQQ/TOMuLE35ACEj8Ow==","content":"

2001年生。硕士研究生。主要研究方向为推荐系统、因果关系推理

"}, bioImg=NoqKQQ/TOMuLE35ACEj8Ow==, bioContent=

2001年生。硕士研究生。主要研究方向为推荐系统、因果关系推理

, aboutCorrespAuthor=null)}, companyList=null), Author(id=1325490369256882272, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, orderNo=3, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325490372662657122, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, authorId=1325490369256882272, language=EN, stringName=Yiting Si, firstName=Yiting, middleName=null, lastName=Si, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=null, bio={"content":"

born in 2001. Master candidate. His main research interests include recommendation system and large language model

"}, bioImg=null, bioContent=

born in 2001. Master candidate. His main research interests include recommendation system and large language model

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325490375623835748, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, authorId=1325490369256882272, language=CN, stringName=司一廷, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=null, bio={"img":"v+zoJ0tHyJb+Zc6Mlpcrwg==","content":"

2001年生。硕士研究生。主要研究方向为推荐系统、大语言模型

"}, bioImg=v+zoJ0tHyJb+Zc6Mlpcrwg==, bioContent=

2001年生。硕士研究生。主要研究方向为推荐系统、大语言模型

, aboutCorrespAuthor=null)}, companyList=null), Author(id=1325490376827600998, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, orderNo=4, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325490377263808615, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, authorId=1325490376827600998, language=EN, stringName=Yusong Ye, firstName=Yusong, middleName=null, lastName=Ye, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=null, bio={"content":"

born in 2001. Master candidate. His main research interests include data mining and recommendation system

"}, bioImg=null, bioContent=

born in 2001. Master candidate. His main research interests include data mining and recommendation system

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325490377746153576, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, authorId=1325490376827600998, language=CN, stringName=叶育松, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=null, bio={"img":"oLR0vp92nEHoQRjKBHejkw==","content":"

2001年生。硕士研究生。主要研究方向为数据挖掘、推荐系统

"}, bioImg=oLR0vp92nEHoQRjKBHejkw==, bioContent=

2001年生。硕士研究生。主要研究方向为数据挖掘、推荐系统

, aboutCorrespAuthor=null)}, companyList=null)], keywords=[Keyword(id=1325490378916364393, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, language=EN, orderNo=1, keyword=multimodal recommendation), Keyword(id=1325490381080625259, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, language=EN, orderNo=2, keyword=feature representations), Keyword(id=1325490381852377196, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, language=EN, orderNo=3, keyword=encoding perspective), Keyword(id=1325490383467184237, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, language=EN, orderNo=4, keyword=hypergraph), Keyword(id=1325490383618179182, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, language=EN, orderNo=5, keyword=cross-modal alignment), Keyword(id=1325490385618862191, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, language=CN, orderNo=1, keyword=多模态推荐), Keyword(id=1325490386025709680, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, language=CN, orderNo=2, keyword=特征表示), Keyword(id=1325490386419974257, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, language=CN, orderNo=3, keyword=编码视角), Keyword(id=1325490386973622386, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, language=CN, orderNo=4, keyword=超图), Keyword(id=1325490387887980659, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, language=CN, orderNo=5, keyword=跨模态对齐)], refs=[Reference(id=1325490405424365710, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=1, rfOrder=0, authorNames=null, journalName=null, refType=null, unstructuredReference=周俊, 董晓蕾, 曹珍富, 等. 推荐系统的隐私保护研究进展[J]. 计算机研究与发展, 2019, 56(10): 2033−2048, articleTitle=null, refAbstract=null), Reference(id=1325490405701189775, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=1, rfOrder=1, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhou Jun, Dong Xiaolei, Cao Zhenfu. Research advances on privacy preserving in recommender systems[J]. Journal of Computer Research and Development, 2019, 56(10): 2033−2048 (in Chinese), articleTitle=null, refAbstract=null), Reference(id=1325490405780881552, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=2, rfOrder=2, authorNames=null, journalName=null, refType=null, unstructuredReference=He Ruining, McAuley J. VBPR: Visual Bayesian personalized ranking from implicit feedback[C] //Proc of the 30th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2016: 144−150, articleTitle=null, refAbstract=null), Reference(id=1325490406179340433, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=3, rfOrder=3, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhang Fuzheng, Lian Defu, Xie Xing, et al. Collaborative knowledge base embedding for recommender systems[C] //Proc of the 22nd ACM SIGKDD Int Conf on Knowledge Discovery and Data Mining. New York: ACM, 2016: 353−362, articleTitle=null, refAbstract=null), Reference(id=1325490408221966482, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=4, rfOrder=4, authorNames=null, journalName=null, refType=null, unstructuredReference=Koren Y, Bell R, Volinsky. Matrix factorization techniques for recommender systems[J]. IEEE Computer, 2009, 42(8): 30−37, articleTitle=null, refAbstract=null), Reference(id=1325490408326824083, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=5, rfOrder=5, authorNames=null, journalName=null, refType=null, unstructuredReference=Chen Lei, Wu Le, Hong Richang, et al. Revisiting graph based collaborative filtering: A linear residual graph convolutional network approach[C] //Proc of the 34th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2020: 27−34, articleTitle=null, refAbstract=null), Reference(id=1325490408662368404, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=6, rfOrder=6, authorNames=null, journalName=null, refType=null, unstructuredReference=Mao Kelong, Zhu Jieming, Xiao Xi, et al. UltraGCN: Ultra simplification of graph convolutional networks for recommendation[C] //Proc of the 30th ACM Int Conf on Information & Knowledge Management. New York: ACM, 2021: 1253−1262, articleTitle=null, refAbstract=null), Reference(id=1325490408746254485, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=7, rfOrder=7, authorNames=null, journalName=null, refType=null, unstructuredReference=Wu Shiwen, Sun Fei, Zhang Wentao, et al. Graph neural networks in recommender systems: A surery[J]. ACM Computing Surveys, 2022, 55(5): 1−37, articleTitle=null, refAbstract=null), Reference(id=1325490409140519062, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=8, rfOrder=8, authorNames=null, journalName=null, refType=null, unstructuredReference=Wei Yinwei, Wang Xiang, He Xiangnan, et al. MMGCN: Multi-modal graph convolution network for personalized recommendation of micro-video[C] // Proc of the 27th ACM Int Conf on Multimedia. New York: ACM, 2019: 1437−1445, articleTitle=null, refAbstract=null), Reference(id=1325490409501229207, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=9, rfOrder=9, authorNames=null, journalName=null, refType=null, unstructuredReference=Wei Yinwei, Wang Xiang, He Xiangnan, et al. Graph-refined convolutional network for multimedia recommendation with implicit feedback[C] //Proc of the 28th ACM Int Conf on Multimedia. New York: ACM, 2020: 3541−3549, articleTitle=null, refAbstract=null), Reference(id=1325490409593503896, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=10, rfOrder=10, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Qifan, Wei Yinwei, Yin Jianhua, et al. DualGNN: Dual graph neural network for multimedia recommendation[J]. IEEE Transactions on Multimedia, 2021, 25(25): 1074−1084, articleTitle=null, refAbstract=null), Reference(id=1325490409895493785, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=11, rfOrder=11, authorNames=null, journalName=null, refType=null, unstructuredReference=Pagliardini M, Gupta P, Jaggi M. Unsupervised learning of sentence embeddings using compositional n-gram features [C] //Proc of the 2018 Conf of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg, PA: ACL, 2018: 528−540, articleTitle=null, refAbstract=null), Reference(id=1325490409970991258, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=12, rfOrder=12, authorNames=null, journalName=null, refType=null, unstructuredReference=Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition[C/OL] //Proc of the 3rd Int Conf on Learning Representations. 2014[2024-05-17]. https://www.semanticscholar.org/reader/eb42cf88027de515750f230b23b1a057dc782108, articleTitle=null, refAbstract=null), Reference(id=1325490410029711515, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=13, rfOrder=13, authorNames=null, journalName=null, refType=null, unstructuredReference=Lin Guojiao, Zhen Meng, Wang Dongjie, et al. GUME: Graphs and user modalities enhancement for long-tail multimodal recommendation[C] //Proc of the 33rd ACM Int Conf on Information and Knowledge Management. New York: ACM, 2024: 1400−1409, articleTitle=null, refAbstract=null), Reference(id=1325490410109403292, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=14, rfOrder=14, authorNames=null, journalName=null, refType=null, unstructuredReference=舒文韬, 李睿潇, 孙天祥, 等. 大型语言模型: 原理、实现与发展[J]. 计算机研究与发展, 2024, 61(2): 351−361, articleTitle=null, refAbstract=null), Reference(id=1325490410386227357, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=14, rfOrder=15, authorNames=null, journalName=null, refType=null, unstructuredReference=Shu Wentao, Li Ruixiao, Sun Tianxiang, et al. Large language models: Principles, implementation, and progress[J]. Journal of Computer Research and Development, 2024, 61(2): 351−361(in Chinese), articleTitle=null, refAbstract=null), Reference(id=1325490410797269150, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=15, rfOrder=16, authorNames=null, journalName=null, refType=null, unstructuredReference=Devlin J, Chang Mingwei, Lee K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding[C] //Proc of the 2019 Conf of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg, PA: ACL, 2019: 4171−4186, articleTitle=null, refAbstract=null), Reference(id=1325490412458213535, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=16, rfOrder=17, authorNames=null, journalName=null, refType=null, unstructuredReference=Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C] //Proc of the 31st Int Conf on Neural Information Processing Systems. Long Beach, CA: NeurIPS, 2017: 6000−6010, articleTitle=null, refAbstract=null), Reference(id=1325490412835700896, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=17, rfOrder=18, authorNames=null, journalName=null, refType=null, unstructuredReference=Wei wei, Ren Xubin, Tang Jiabin, et al. LLMRec: Large language models with graph augmentation for recommendation[C] //Proc pf the 17th ACM Int Conf on Web Search and Data Mining. New York: ACM, 2024: 806−815, articleTitle=null, refAbstract=null), Reference(id=1325490412948947105, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=18, rfOrder=19, authorNames=null, journalName=null, refType=null, unstructuredReference=Tan Juntao, Xu Shuyuan, Hua Yingqiang, et al. IDGenRec: LLM-RecSys alignment with textual id learning[C] //Proc of the 47th Int ACM Conf on Research and Development in Information Retrieval. New York: ACM, 2024: 355−364, articleTitle=null, refAbstract=null), Reference(id=1325490413271908514, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=19, rfOrder=20, authorNames=null, journalName=null, refType=null, unstructuredReference=Bao Keqin, Zhang Jizhi, Zhang Yang, et al. TALLRec: An effectiveand efficient tuning framework to align large language model with recommendation[C] //Proc of the 17th ACM Conf on Recommender Systems. New York: ACM, 2023: 1007−1014, articleTitle=null, refAbstract=null), Reference(id=1325490413347405987, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=20, rfOrder=21, authorNames=null, journalName=null, refType=null, unstructuredReference=Wang Suhang, Wang Yilin, Tang Jiliang, et al. What your images reveal: Exploiting visual contents for point-of-interest recommendation[C] //Proc of the 26th Int Conf on World Wide Web. New York: ACM, 2017: 391−400, articleTitle=null, refAbstract=null), Reference(id=1325490413703921828, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=21, rfOrder=22, authorNames=null, journalName=null, refType=null, unstructuredReference=Chen Jingyuan, Zhang Hanwang, He Xiangnan, et al. Attentive collaborative filtering: Multimedia recommendation with item-and component-level attention[C] //Proc of the 40th Int ACM SIGIR Conf on Research and Development in Information Retrieval. New York: ACM, 2017: 335−344, articleTitle=null, refAbstract=null), Reference(id=1325490414089797797, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=22, rfOrder=23, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhang Jinghao, Zhu Yanqiao, Liu Qiang, et al. Mining latent structures for multimedia recommendation[C] //Proc of the 29th ACM Int Conf on Multimedia. New York: ACM, 2021: 3872−3880, articleTitle=null, refAbstract=null), Reference(id=1325490414156906662, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=23, rfOrder=24, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhou Xin, Shen Zhiqi. A tale of two graphs: Freezing and denoising graph structures for multimodal recommendation[C] //Proc of the 31st ACM Int Conf on Multimedia. New York: ACM, 2023: 935−943, articleTitle=null, refAbstract=null), Reference(id=1325490414215626919, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=24, rfOrder=25, authorNames=null, journalName=null, refType=null, unstructuredReference=Sun Rui, Cao Xuezhi, Zhao Yan, et al. Multi-modal knowledge graphs for recommender systems[C] //Proc of the 29th ACM Int Conf on Information & Knowledge Management. New York: ACM, 2020: 1405−1414, articleTitle=null, refAbstract=null), Reference(id=1325490414278541480, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=25, rfOrder=26, authorNames=null, journalName=null, refType=null, unstructuredReference=林晶晶, 冶忠林, 赵海兴, 等. 超图神经网络综述[J]. 计算机研究与发展, 2024, 61(2): 362−384, articleTitle=null, refAbstract=null), Reference(id=1325490414526005417, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=25, rfOrder=27, authorNames=null, journalName=null, refType=null, unstructuredReference=Lin Jingjing, Ye Zhonglin, Zhao Haixing, et al. Survey on hypergraph nerual networks[J]. Journal of Computer Research and Development, 2024, 61(2): 362−384(in Chinese), articleTitle=null, refAbstract=null), Reference(id=1325490414601502890, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=26, rfOrder=28, authorNames=null, journalName=null, refType=null, unstructuredReference=Feng Yifan, You Haoxuan, Zhang Zizhao, et al. Hypergraph neural networks[C] //Proc of the 33rd AAAI Conf on Artificial Intelligence and 31st Innovative Applications of Artificial Intelligence Conf and 9th AAAI Symp on Educational Advances in Artificial Intelligence. Palo Alto, CA: AAAI, 2019: 3558−3565, articleTitle=null, refAbstract=null), Reference(id=1325490414672806059, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=27, rfOrder=29, authorNames=null, journalName=null, refType=null, unstructuredReference=Jang E, Gu Shixiang, Poole B. Categorical reparameterization with Gumbel-Softmax[C/OL] //Proc of the 5th Int Conf Learning Representations. 2016[2023-03-19]. https://www.researchgate.net/publication/309663606_Categorical_Reparameterization_with_Gumbel-Softmax, articleTitle=null, refAbstract=null), Reference(id=1325490415025127596, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=28, rfOrder=30, authorNames=null, journalName=null, refType=null, unstructuredReference=Xia Lianghao, Huang Chao, Zhang Chuxu. Self-supervised hypergraph transformer for recommender systems[C] //Proc of the 28th ACM SIGKDD Conf on Knowledge Discovery and Data Mining. New York: ACM, 2022: 2100−2109, articleTitle=null, refAbstract=null), Reference(id=1325490415356477613, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=29, rfOrder=31, authorNames=null, journalName=null, refType=null, unstructuredReference=Guo Zhiqiang, Li Jianjun, Li Guohui, et al. LGMRec: Local and global graph learning for multimodal recommendation[C] //Proc of the 38th AAAI Conf on Artificial Intelligence and 36th Conf on Innovative Applications of Artificial Intelligence and 14th Symp on Educational Advances in Artificial Intelligence. Palo Alto, CA: AAAI, 2024: 8454−8462, articleTitle=null, refAbstract=null), Reference(id=1325490416853844142, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=30, rfOrder=32, authorNames=null, journalName=null, refType=null, unstructuredReference=Xu Jinfeng, Chen Zheyu, Yang Shuo, et al. MENTOR: Multi-level self-supervised learning for multimodal recommendation[C] //Proc of the 39th Annual AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2025: 12908−12917, articleTitle=null, refAbstract=null), Reference(id=1325490416933535919, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=31, rfOrder=33, authorNames=null, journalName=null, refType=null, unstructuredReference=McAuley J, Targett C, Shi Qinfeng, et al. Image-based recommendations on styles and substitutes[C] //Proc of the 38th Int ACM SIGIR Conf on Research and Development in Information Retrieval. New York: ACM, 2015: 43−52, articleTitle=null, refAbstract=null), Reference(id=1325490417004839088, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=32, rfOrder=34, authorNames=null, journalName=null, refType=null, unstructuredReference=Rendle S, Freudenthaler C, Gantner Z, et al. BPR: Bayesian personalized ranking from implicit feedback[C] //Proc of the 25th Conf on Uncertainty in Artificial Intelligence. Palo Alto, CA: AUAI, 2009: 452−461, articleTitle=null, refAbstract=null), Reference(id=1325490417101308081, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=33, rfOrder=35, authorNames=null, journalName=null, refType=null, unstructuredReference=He Xiangnan, Deng Kuan, Wang Xiang, et al. LightGCN: Simplifying and powering graph convolution network for recommendation[C] //Proc of the 43rd Int ACM SIGIR Conf on Research and Development in Information Retrieval. New York: ACM, 2020: 639−648, articleTitle=null, refAbstract=null), Reference(id=1325490417185194162, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=34, rfOrder=36, authorNames=null, journalName=null, refType=null, unstructuredReference=Lin Zihan, Tian Changxin, Hou Yupeng, et al. Improving graph collaborative filtering with neighborhood-enriched contrastive learning[C] //Proc of the ACM Web Conf. New York: ACM, 2022: 2320−2329, articleTitle=null, refAbstract=null), Reference(id=1325490417269080243, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=35, rfOrder=37, authorNames=null, journalName=null, refType=null, unstructuredReference=Yi Zixuan, Wang Xi, Ounis I, et al. Multi-modal graph contrastive learning for micro-video recommendation[C] //Proc of the 45th Int ACM SIGIR Conf on Research and Development in Information Retrieval. New York: ACM, 2022: 1807−1811, articleTitle=null, refAbstract=null), Reference(id=1325490417361354932, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=36, rfOrder=38, authorNames=null, journalName=null, refType=null, unstructuredReference=Tao Zhulin, Liu Xiaohao, Xia Yewei, et al. Self-supervised learning formultimedia recommendation[J]. IEEE Transactions on Multimedia, 2022, 25(25): 5107−5116, articleTitle=null, refAbstract=null), Reference(id=1325490417466212533, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=37, rfOrder=39, authorNames=null, journalName=null, refType=null, unstructuredReference=Zhou Xin, Zhou Hongyu, Liu Yong, et al. Bootstrap latent representations for multi-modal recommendation[C] //Proc of the ACM Web Conf. New York: ACM, 2023: 845−854, articleTitle=null, refAbstract=null), Reference(id=1325490417558487222, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=38, rfOrder=40, authorNames=null, journalName=null, refType=null, unstructuredReference=Jiang Yangqin, Xia Lianghao, Wei Wei, et al. DiffMM: Multi-modal diffusion model for recommendation[C] //Proc of the 32nd ACM Conf on Multimedia. New York: ACM, 2024: 7591−7599, articleTitle=null, refAbstract=null), Reference(id=1325490417654956215, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=39, rfOrder=41, authorNames=null, journalName=null, refType=null, unstructuredReference=Glorot X, Bengio Y. Understanding the difficulty of training deep feedforward neural networks[C] //Proc of the 13th Int Conf on Artificial Intelligence and Statistics. Brookline, MA: JMLR. org, 2010: 249−256, articleTitle=null, refAbstract=null), Reference(id=1325490417793368248, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, doi=null, pmid=null, pmcid=null, year=null, volume=null, issue=null, pageStart=null, pageEnd=null, url=null, language=null, rfNumber=40, rfOrder=42, authorNames=null, journalName=null, refType=null, unstructuredReference=Kingma D, Ba J. Adam: A method for stochastic optimization[C/OL] //Proc of the 3rd Int Conf on Learning Representations. 2014[2024-04-12]. https://www.semanticscholar.org/reader/a6cb366736791bcccc5c8639de5a8f9636bf87e8, articleTitle=null, refAbstract=null)], funds=null, companyList=[AuthorCompany(id=1325490354610372684, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, xref=null, ext=[AuthorCompanyExt(id=1325490355000442957, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, companyId=1325490354610372684, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=School of Computer Science and Engineering, Northeastern University, Shenyang 110169), AuthorCompanyExt(id=1325490355390513230, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, companyId=1325490354610372684, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=东北大学计算机科学与工程学院 沈阳 110169)]), AuthorCompany(id=1325490357139538000, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, xref=null, ext=[AuthorCompanyExt(id=1325490357164703825, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, companyId=1325490357139538000, language=EN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=Key Laboratory of Intelligent Computing in Medical Image (Northeastern University), Ministry of Education, Shenyang 110169), AuthorCompanyExt(id=1325490357982593106, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, companyId=1325490357139538000, language=CN, country=null, province=null, city=null, postcode=null, companyName=null, departmentName=null, remark=医学影像智能计算教育部重点实验室(东北大学) 沈阳 110169)])], figs=[ArticleFig(id=1325490388978499700, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, language=EN, label=Figure 1, caption=A brief schematic diagram of multimodal recommendation, figureFileSmall=vxfKfMDCUho5Wmnxgka5zA==, figureFileBig=KiLe02Huc1uETDAg520Sig==, tableContent=null), ArticleFig(id=1325490390203236469, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, language=CN, label=图1, caption=多模态推荐简要示意图, figureFileSmall=vxfKfMDCUho5Wmnxgka5zA==, figureFileBig=KiLe02Huc1uETDAg520Sig==, tableContent=null), ArticleFig(id=1325490392283611254, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, language=EN, label=Figure 2, caption=Clothing recommendation example, figureFileSmall=+clMaTBnYtuunYnQRO4bxw==, figureFileBig=YGtW8dLNJhYlHApQgS5Xgg==, tableContent=null), ArticleFig(id=1325490394401734776, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, language=CN, label=图2, caption=衣服推荐示例, figureFileSmall=+clMaTBnYtuunYnQRO4bxw==, figureFileBig=YGtW8dLNJhYlHApQgS5Xgg==, tableContent=null), ArticleFig(id=1325490394531758201, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, language=EN, label=Figure 3, caption=Overall framework diagram of system, figureFileSmall=jh1Xdi1T+JuGVgDcw7qAjw==, figureFileBig=oihnNcT0SOQKLhSOza5ovw==, tableContent=null), ArticleFig(id=1325490394896662650, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, language=CN, label=图3, caption=系统整体框架图, figureFileSmall=jh1Xdi1T+JuGVgDcw7qAjw==, figureFileBig=oihnNcT0SOQKLhSOza5ovw==, tableContent=null), ArticleFig(id=1325490395269955707, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, language=EN, label=Figure 4, caption=Multimodal encoding layer, figureFileSmall=yIC3WKCQNPWZDgc3icClXA==, figureFileBig=/iMt50mbc9UXq+sdn+ujSQ==, tableContent=null), ArticleFig(id=1325490395680997500, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, language=CN, label=图4, caption=多模态编码层, figureFileSmall=yIC3WKCQNPWZDgc3icClXA==, figureFileBig=/iMt50mbc9UXq+sdn+ujSQ==, tableContent=null), ArticleFig(id=1325490395752300669, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, language=EN, label=Figure 5, caption=Multimodal alignment, figureFileSmall=TiSngDZ9qHrlQiS7/S1eLg==, figureFileBig=H4F9BScwii3Lamp23XB5nQ==, tableContent=null), ArticleFig(id=1325490396469526654, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, language=CN, label=图5, caption=多模态对齐, figureFileSmall=TiSngDZ9qHrlQiS7/S1eLg==, figureFileBig=H4F9BScwii3Lamp23XB5nQ==, tableContent=null), ArticleFig(id=1325490396549218431, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, language=EN, label=Figure 6, caption=Performance comparison under neighbor $ K $ in different datasets, figureFileSmall=S6uA1DVv7fVTDePS9rwptQ==, figureFileBig=/+QS3lyXBmYyElS1dnNHlg==, tableContent=null), ArticleFig(id=1325490396935094400, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, language=CN, label=图6, caption=不同数据集的邻居$ K $下的性能比较, figureFileSmall=S6uA1DVv7fVTDePS9rwptQ==, figureFileBig=/+QS3lyXBmYyElS1dnNHlg==, tableContent=null), ArticleFig(id=1325490397752983682, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, language=EN, label=Figure 7, caption=Effect of the balancing hyper-parameter $ {\lambda }_{\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{g}\mathrm{n}} $, figureFileSmall=verG4qWit7r+Kzbkn6wIDw==, figureFileBig=tSXNYCBirDtbufiVul7StA==, tableContent=null), ArticleFig(id=1325490399049023619, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, language=CN, label=图7, caption=平衡超参数$ {\lambda }_{\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{g}\mathrm{n}} $的影响, figureFileSmall=verG4qWit7r+Kzbkn6wIDw==, figureFileBig=tSXNYCBirDtbufiVul7StA==, tableContent=null), ArticleFig(id=1325490399162269828, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, language=EN, label=Figure 8, caption=Performance of models on Baby dataset with different $ N $ and $ \alpha $ settings, figureFileSmall=czk8ZtQ9kLTI38QIdK9w+g==, figureFileBig=qxQv/0yxFX7Ir3zLAXCHcg==, tableContent=null), ArticleFig(id=1325490399497814149, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, language=CN, label=图8, caption=Baby数据集上的不同$ N $和$ \alpha $设置下模型的性能, figureFileSmall=czk8ZtQ9kLTI38QIdK9w+g==, figureFileBig=qxQv/0yxFX7Ir3zLAXCHcg==, tableContent=null), ArticleFig(id=1325490399841747078, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, language=EN, label=Table 1, caption=Description of the Main Notations, figureFileSmall=null, figureFileBig=null, tableContent=
符号描述
$ U=\left\{u\right\} $用户集
$ I=\left\{i\right\} $项目集
$ M $模态集合
$ m $模态
$ G $二分图
$ V $节点集
$ E $边集
$ d $ID嵌入维度
$ {d}^{m} $LLM处理模态嵌入的维度
$ \boldsymbol{R}\in {\left\{\mathrm{0,1}\right\}}^{\left|U\right|\times \left|I\right|} $交互矩阵
$ \boldsymbol{A}\in {\mathbb{R}}^{\left(\left|U\right|+\left|I\right|\right)\times \left(\left|U\right|+\left|I\right|\right)} $根据交互矩阵R构造的邻接矩阵
$ {\boldsymbol{E}}_{i,m} $LLM处理的模态嵌入
$ {\boldsymbol{E}}_{\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{t}\mathrm{y}} $LLM处理的实体词嵌入
$ {\boldsymbol{E}}^{m,{H}} $超图得到的模态嵌入
$ {\boldsymbol{E}}^{{H}} $超图得到的混合模态嵌入
$ {\boldsymbol{V}}^{m} $模态超边初始化嵌入
$ {\boldsymbol{E}}_{m\_\mathrm{r}\mathrm{e}\mathrm{p}} $记忆视角得到的模态嵌入
$ {\boldsymbol{E}}_{m}^{ \sim } $扩展图得到的模态嵌入
$ {\boldsymbol{E}}_{u,{\mathrm{id}}}^{ \wedge } $扩展视角输入用户ID初始化嵌入
$ {\boldsymbol{E}}_{i,{\mathrm{id}}}^{\wedge} $扩展视角输入项目ID初始化嵌入
$ {\boldsymbol{E}}_{i,m}^{+} $项目同质图得到的模态嵌入
$ {\boldsymbol{E}}_{u,m} $用户模态初始化嵌入
$ {\boldsymbol{H}}_{i}^{m} $项目超边依赖矩阵
$ {\boldsymbol{H}}_{u}^{m} $用户超边依赖矩阵
), ArticleFig(id=1325490399950798983, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, language=CN, label=表1, caption=

主要符号的描述

, figureFileSmall=null, figureFileBig=null, tableContent=
符号描述
$ U=\left\{u\right\} $用户集
$ I=\left\{i\right\} $项目集
$ M $模态集合
$ m $模态
$ G $二分图
$ V $节点集
$ E $边集
$ d $ID嵌入维度
$ {d}^{m} $LLM处理模态嵌入的维度
$ \boldsymbol{R}\in {\left\{\mathrm{0,1}\right\}}^{\left|U\right|\times \left|I\right|} $交互矩阵
$ \boldsymbol{A}\in {\mathbb{R}}^{\left(\left|U\right|+\left|I\right|\right)\times \left(\left|U\right|+\left|I\right|\right)} $根据交互矩阵R构造的邻接矩阵
$ {\boldsymbol{E}}_{i,m} $LLM处理的模态嵌入
$ {\boldsymbol{E}}_{\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{t}\mathrm{y}} $LLM处理的实体词嵌入
$ {\boldsymbol{E}}^{m,{H}} $超图得到的模态嵌入
$ {\boldsymbol{E}}^{{H}} $超图得到的混合模态嵌入
$ {\boldsymbol{V}}^{m} $模态超边初始化嵌入
$ {\boldsymbol{E}}_{m\_\mathrm{r}\mathrm{e}\mathrm{p}} $记忆视角得到的模态嵌入
$ {\boldsymbol{E}}_{m}^{ \sim } $扩展图得到的模态嵌入
$ {\boldsymbol{E}}_{u,{\mathrm{id}}}^{ \wedge } $扩展视角输入用户ID初始化嵌入
$ {\boldsymbol{E}}_{i,{\mathrm{id}}}^{\wedge} $扩展视角输入项目ID初始化嵌入
$ {\boldsymbol{E}}_{i,m}^{+} $项目同质图得到的模态嵌入
$ {\boldsymbol{E}}_{u,m} $用户模态初始化嵌入
$ {\boldsymbol{H}}_{i}^{m} $项目超边依赖矩阵
$ {\boldsymbol{H}}_{u}^{m} $用户超边依赖矩阵
), ArticleFig(id=1325490401519468680, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, language=EN, label=Table 2, caption=Statistics of Datasets, figureFileSmall=null, figureFileBig=null, tableContent=
数据集用户数量项目数量交互数量稀疏性/%
Baby19445705016079299.88
Sports355981835729633799.95
Clothing393872303327867799.97
), ArticleFig(id=1325490402001813641, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, language=CN, label=表2, caption=

数据集统计

, figureFileSmall=null, figureFileBig=null, tableContent=
数据集用户数量项目数量交互数量稀疏性/%
Baby19445705016079299.88
Sports355981835729633799.95
Clothing393872303327867799.97
), ArticleFig(id=1325490403679535242, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, language=EN, label=Table 3, caption=Overall Performance Comparison of the Models, figureFileSmall=null, figureFileBig=null, tableContent=
模型BabySportsClothing
R@10R@20N@10N@20R@10R@20N@10N@20R@10R@20N@10N@20
注:R@10,R@20,N@10,N@20分别代表Recall@10,Recall@20,NDCG@10,NDCG@20;黑体数值表示本文模型达到最好的实验效果。
MF-BPR0.03570.05750.01920.02490.04320.06530.02410.02980.01870.02790.01030.0126
LightGCN0.04790.07540.02570.03280.05690.08640.03130.03870.03400.05260.01880.0236
NCL0.05380.08360.02920.03690.06160.09400.03390.04210.04100.06070.02280.0275
VBPR0.04230.06630.02230.02840.05580.08560.03070.03840.02810.04150.01580.0192
MMGCN0.03780.06150.02000.02610.03700.06050.01930.02540.03780.06150.02000.0247
LATTICE0.05470.08500.02920.03700.06200.09530.03350.04210.04920.07330.02680.0330
MMGCL0.05220.07780.02890.03550.06600.09940.03620.04480.04380.06690.02390.0297
SLMRec0.05400.08100.02850.03570.06760.10170.03740.04620.05400.08100.02850.0357
BM30.05640.08830.03010.03830.06560.09800.03550.04380.04220.06210.02310.0281
DiffMM0.06240.09750.03230.04110.06790.10170.03710.04580.05120.07930.02760.0325
MENTOR0.06530.10360.03490.04470.07370.10950.03960.04970.06090.08980.03320.0402
LGMRec0.06440.10020.03490.04400.07200.10680.03900.04800.05550.08280.03020.0371
IDGenRec0.06510.10350.03470.04430.06830.10620.03720.04750.05930.08790.03270.0391
DPRec(本文)0.06730.10450.03570.04520.07120.10830.03870.04830.06160.09030.03340.0407
), ArticleFig(id=1325490404111548555, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, language=CN, label=表3, caption=

模型的整体性能比较

, figureFileSmall=null, figureFileBig=null, tableContent=
模型BabySportsClothing
R@10R@20N@10N@20R@10R@20N@10N@20R@10R@20N@10N@20
注:R@10,R@20,N@10,N@20分别代表Recall@10,Recall@20,NDCG@10,NDCG@20;黑体数值表示本文模型达到最好的实验效果。
MF-BPR0.03570.05750.01920.02490.04320.06530.02410.02980.01870.02790.01030.0126
LightGCN0.04790.07540.02570.03280.05690.08640.03130.03870.03400.05260.01880.0236
NCL0.05380.08360.02920.03690.06160.09400.03390.04210.04100.06070.02280.0275
VBPR0.04230.06630.02230.02840.05580.08560.03070.03840.02810.04150.01580.0192
MMGCN0.03780.06150.02000.02610.03700.06050.01930.02540.03780.06150.02000.0247
LATTICE0.05470.08500.02920.03700.06200.09530.03350.04210.04920.07330.02680.0330
MMGCL0.05220.07780.02890.03550.06600.09940.03620.04480.04380.06690.02390.0297
SLMRec0.05400.08100.02850.03570.06760.10170.03740.04620.05400.08100.02850.0357
BM30.05640.08830.03010.03830.06560.09800.03550.04380.04220.06210.02310.0281
DiffMM0.06240.09750.03230.04110.06790.10170.03710.04580.05120.07930.02760.0325
MENTOR0.06530.10360.03490.04470.07370.10950.03960.04970.06090.08980.03320.0402
LGMRec0.06440.10020.03490.04400.07200.10680.03900.04800.05550.08280.03020.0371
IDGenRec0.06510.10350.03470.04430.06830.10620.03720.04750.05930.08790.03270.0391
DPRec(本文)0.06730.10450.03570.04520.07120.10830.03870.04830.06160.09030.03340.0407
), ArticleFig(id=1325490404908466316, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, language=EN, label=Table 4, caption=Effect of Model Performance of Each Module, figureFileSmall=null, figureFileBig=null, tableContent=
数据集BabySportsClothing
R@10R@20N@10N@20R@10R@20N@10N@20R@10R@20N@10N@20
注:R@10,R@20,N@10,N@20分别代表Recall@10,Recall@20,NDCG@10,NDCG@20。DPRec-o表示使用原始嵌入替换掉由LLM处理得到的嵌入;DPRec-e表示在编码层舍弃扩展视角编码;DPRec-m表示在编码层舍弃记忆视角编码;DPRec-c表示去掉超图跨模态对齐操作;DPRec-c1表示保留前2个级别的跨模态对齐,去掉第3个级别的跨模态对齐操作。
DPRec0.06730.10450.03570.04520.07120.10830.03870.04830.06160.09030.03340.0407
DPRec-o0.06190.10160.03360.04210.06340.10370.03560.04530.05260.08120.02750.0366
DPRec-e0.06600.10200.03510.04490.06710.10420.03620.04530.05600.08430.02970.0375
DPRec-m0.02350.03820.01250.01670.03170.04620.02100.02530.01220.02760.01130.0195
DPRec-c0.06650.10390.03510.04460.06950.10750.03790.04750.05920.08850.03250.0389
DPRec-c10.06690.10410.03540.04490.07040.10790.03820.04800.06010.08960.03300.0392
), ArticleFig(id=1325490405306925197, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, language=CN, label=表4, caption=

各个模块的模型性能影响

, figureFileSmall=null, figureFileBig=null, tableContent=
数据集BabySportsClothing
R@10R@20N@10N@20R@10R@20N@10N@20R@10R@20N@10N@20
注:R@10,R@20,N@10,N@20分别代表Recall@10,Recall@20,NDCG@10,NDCG@20。DPRec-o表示使用原始嵌入替换掉由LLM处理得到的嵌入;DPRec-e表示在编码层舍弃扩展视角编码;DPRec-m表示在编码层舍弃记忆视角编码;DPRec-c表示去掉超图跨模态对齐操作;DPRec-c1表示保留前2个级别的跨模态对齐,去掉第3个级别的跨模态对齐操作。
DPRec0.06730.10450.03570.04520.07120.10830.03870.04830.06160.09030.03340.0407
DPRec-o0.06190.10160.03360.04210.06340.10370.03560.04530.05260.08120.02750.0366
DPRec-e0.06600.10200.03510.04490.06710.10420.03620.04530.05600.08430.02970.0375
DPRec-m0.02350.03820.01250.01670.03170.04620.02100.02530.01220.02760.01130.0195
DPRec-c0.06650.10390.03510.04460.06950.10750.03790.04750.05920.08850.03250.0389
DPRec-c10.06690.10410.03540.04490.07040.10790.03820.04800.06010.08960.03300.0392
)], attaches=null, journal=Journal(id=1324759606947315908, delFlag=0, nameCn=计算机研究与发展, nameEn=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, issn=1000-1239, eissn=null, cn=11-1777/TP, coden=null, periodic=0, language=CN, oaType=null, ccby=null, superviseOffice=null, ownerOffice=null, pubOffice=null, editorOffice=null, officeType=null, aims=null, clcCode=null, officeProv=null, officeCity=null, officeAddr=null, officeZip=null, officeEmail=null, officePhone=null, editDirector=null, officeDirector=null, officeDirectorPhone=null, officeStaffNum=null, officeEmpNum=null, coverPicUrl=k2l8zgqwstuxxevFX2qxVA==, journalPrice=null, startedYear=null, abbrevIsoEn=Journal of Computer Research and Development, journalRemark=null, publicationField=null, createdTime=1793776903141, updatedTime=1793850698612, createdBy=18614031015, updatedBy=13701087609, firstLetterCn=J, firstLetterEn=J, subjectCode=Electronic Information, subjectName=null, subjectCodeEn=Electronic Information, subjectNameEn=null, picCn=k2l8zgqwstuxxevFX2qxVA==, picEn=r6k0Ry+RD+c/VXj5QarmDw==, jcr=null, cjcr=null, exts=[JournalExt(id=1325069127710286688, language=CN, name=计算机研究与发展, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698640, updatedTime=1793850698640, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""}), JournalExt(id=1325069127777395553, language=EN, name=Journal of Computer Research and Development, nameHistory1=null, nameHistory2=null, managedBy=, sponsoredBy=, publishedBy=, editorOffice=, officeProv=null, officeCity=null, officeAddr=, officeZip=, editDirector=, officeDirector=null, officePhone=null, coverPicUrl=null, journalRemark=, submitArticleUrl=null, websiteUrl=, createdTime=1793850698656, updatedTime=1793850698656, createdBy=13701087609, updatedBy=13701087609, submissionGuidelinesUrl=, submissionAuthorUrl=https://crad.ict.ac.cn/journalx/authorLogOn.action, submissionEditorUrl=https://crad.ict.ac.cn/journalx/editorLogOn.action, submissionReviewUrl=https://crad.ict.ac.cn/journalx/expertLogOn.action, submissionCeEditorUrl=https://crad.ict.ac.cn/journalx/editorInChiefLogOn.action, submissionAeEditorUrl=, option={"copyright":""})], databaseList=null, tenantJournalId=1324759914146529478, websiteList=[Website(id=1325070300139892771, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN, language=CN, createTime=1793850978169, createBy=18614031015, updateTime=1793851040259, updateBy=18614031015, name=计算机研究与发展-中文, tplId=1146099689490845704, title=计算机研究与发展, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070893751349360, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=articleTextType, value=kx, createTime=1793851119697, updateTime=1793851119697, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893730377837, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=banner, value=null, createTime=1793851119692, updateTime=1793851119692, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893772320883, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=grayFlag, value=0, createTime=1793851119702, updateTime=1793851119702, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893721989228, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851119690, updateTime=1793851119690, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893784903797, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=minRunFlag, value=0, createTime=1793851119705, updateTime=1793851119705, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893742960751, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/file/pic, createTime=1793851119695, updateTime=1793851119695, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893780709492, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=silenceFlag, value=0, createTime=1793851119704, updateTime=1793851119704, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893738766446, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_cn_619/, createTime=1793851119694, updateTime=1793851119694, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893755543665, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeColor, value=null, createTime=1793851119698, updateTime=1793851119698, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070893763932274, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300139892771, code=themeStyle, value=null, createTime=1793851119700, updateTime=1793851119700, creator=18614031015, updator=18614031015)]), Website(id=1325070300580294698, webName=null, webTitle=null, webDomain=null, webCopyrigh=null, webIpcNo=null, seoTitle=null, seoKeywords=null, seoDescription=null, tenantJournalId=null, journalId=1324759914146529478, journalNameCn=null, journalNameEn=null, grayFlag=null, tenantId=1146029695717560320, platformId=null, journalGroupId=null, journalGroupNameCn=null, journalGroupNameEn=null, type=1, domain=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN, language=EN, createTime=1793850978274, createBy=18614031015, updateTime=1793851036638, updateBy=18614031015, name=计算机研究与发展-英文, tplId=1146101810881728533, title=Journal of Computer Research and Development, delFlag=0, indexPage=/home, props=[WebsiteProps(id=1325070925288321148, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=articleTextType, value=kx, createTime=1793851127216, updateTime=1793851127216, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925263155321, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=banner, value=null, createTime=1793851127210, updateTime=1793851127210, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925309292671, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=grayFlag, value=0, createTime=1793851127221, updateTime=1793851127221, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925254766712, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=logo, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic?fileId=U27tDsy717YF7XSPA+Vwdw==, createTime=1793851127208, updateTime=1793851127208, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925326069889, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=minRunFlag, value=0, createTime=1793851127225, updateTime=1793851127225, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925279932539, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=picServerUrl, value=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/file/pic, createTime=1793851127214, updateTime=1793851127214, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925317681280, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=silenceFlag, value=0, createTime=1793851127223, updateTime=1793851127223, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925271543930, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=staticResourcePath, value=https://castjournals.cast.org.cn/joweb/cast_kjdb_en_623/, createTime=1793851127212, updateTime=1793851127212, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925292515453, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeColor, value=null, createTime=1793851127217, updateTime=1793851127217, creator=18614031015, updator=18614031015), WebsiteProps(id=1325070925300904062, tenantId=1146029695717560320, journalId=null, journalGroupId=null, siteId=1325070300580294698, code=themeStyle, value=null, createTime=1793851127219, updateTime=1793851127219, creator=18614031015, updator=18614031015)])], journalTitle=计算机研究与发展, weixinUrl=null, journalUrl=https://crad.ict.ac.cn/, iacademicId=null, status=1, seqNo=null, journalTitleEn=Journal of Computer Research and Development, journalPhotoCn=k2l8zgqwstuxxevFX2qxVA==, journalPhotoEn=r6k0Ry+RD+c/VXj5QarmDw==, journalFirstLetter=J, journalRecommend=null, journalNew=null, journalCollection=null, jcrJf=null, cjcrJf=null, jcrJfStr=null, cjcrJfStr=null, submissionFirstDecision=null, sciSubjectClassification=null, casSubjectClassification=null, citeScore=null, totalCitationFrequency=null, icpCode=null, psCode=null, advertisingLicenseCode=null, copyrightInformation=null, country=null, option=, provinceCode=null, provinceName=null, collectFlag=false, interPubPlatform=, interPubPlatformUrl=null), detailUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202550040, detailUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202550040, pdfUrlCn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/PDF/10.7544/issn1000-1239.202550040, pdfUrlEn=https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/PDF/10.7544/issn1000-1239.202550040, aliStartDate=0, aliEndDate=0, collectionFlag=false, citedCount=null, citedUrl=null, previewStatus=0, delFlag=0, hasFullText=1, orderTime=1767196800000, fullTextJson=null, articleText=null, reference=null)
收藏切换
基于大语言模型的双视角多级跨模态推荐
李佚名 , 于亚新 * , 于之晟 , 司一廷 , 叶育松
计算机研究与发展 | 人工智能 2026,63(1): 147-161
收起
收藏切换
计算机研究与发展 |人工智能 2026 , 63 (1) : 147 -161
基于大语言模型的双视角多级跨模态推荐
全屏
[Author(id=1325490359236690004, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, orderNo=0, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=2372016@stu.neu.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325490359647731797, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, authorId=1325490359236690004, language=EN, stringName=Yiming Li, firstName=Yiming, middleName=null, lastName=Li, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=null, bio={"content":"

born in 2000. Master candidate. His main research interests include knowledge graph and recommendation system

"}, bioImg=null, bioContent=

born in 2000. Master candidate. His main research interests include knowledge graph and recommendation system

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325490359979081814, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, authorId=1325490359236690004, language=CN, stringName=李佚名, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=null, bio={"img":"TZl6+eeluZ2NnrbxcqrDnA==","content":"

2000年生。硕士研究生。主要研究方向为知识图谱、推荐系统

"}, bioImg=TZl6+eeluZ2NnrbxcqrDnA==, bioContent=

2000年生。硕士研究生。主要研究方向为知识图谱、推荐系统

, aboutCorrespAuthor=null)}, companyList=null), Author(id=1325490360482398296, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, orderNo=1, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=yuyx@mail.neu.edu.cn, emailSecond=null, emailThird=null, correspondingAuthor=1, authorType=1, ext={EN=AuthorExt(id=1325490363779121241, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, authorId=1325490360482398296, language=EN, stringName=Yaxin Yu, firstName=Yaxin, middleName=null, lastName=Yu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=*, address=null, bio={"content":"

born in 1971. PhD, associate professor, master supervisor. Member of IEEE, ACM and CCF. Her main research interests include data mining and social network

"}, bioImg=null, bioContent=

born in 1971. PhD, associate professor, master supervisor. Member of IEEE, ACM and CCF. Her main research interests include data mining and social network

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325490365930799194, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, authorId=1325490360482398296, language=CN, stringName=于亚新, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=*, address=null, bio={"img":"5bQINdYT1R3SJn2G1c3uew==","content":"

1971年生。博士,副教授,硕士生导师。IEEE,ACM,CCF会员。主要研究方向为数据挖掘、社交网络

"}, bioImg=5bQINdYT1R3SJn2G1c3uew==, bioContent=

1971年生。博士,副教授,硕士生导师。IEEE,ACM,CCF会员。主要研究方向为数据挖掘、社交网络

, aboutCorrespAuthor=null)}, companyList=null), Author(id=1325490366782242908, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, orderNo=2, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325490367973425245, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, authorId=1325490366782242908, language=EN, stringName=Zhisheng Yu, firstName=Zhisheng, middleName=null, lastName=Yu, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=null, bio={"content":"

born in 2001. Master candidate. His main research interests include recommendation system and causal relationship inference

"}, bioImg=null, bioContent=

born in 2001. Master candidate. His main research interests include recommendation system and causal relationship inference

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325490368501907550, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, authorId=1325490366782242908, language=CN, stringName=于之晟, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=null, bio={"img":"NoqKQQ/TOMuLE35ACEj8Ow==","content":"

2001年生。硕士研究生。主要研究方向为推荐系统、因果关系推理

"}, bioImg=NoqKQQ/TOMuLE35ACEj8Ow==, bioContent=

2001年生。硕士研究生。主要研究方向为推荐系统、因果关系推理

, aboutCorrespAuthor=null)}, companyList=null), Author(id=1325490369256882272, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, orderNo=3, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325490372662657122, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, authorId=1325490369256882272, language=EN, stringName=Yiting Si, firstName=Yiting, middleName=null, lastName=Si, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=null, bio={"content":"

born in 2001. Master candidate. His main research interests include recommendation system and large language model

"}, bioImg=null, bioContent=

born in 2001. Master candidate. His main research interests include recommendation system and large language model

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325490375623835748, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, authorId=1325490369256882272, language=CN, stringName=司一廷, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=null, bio={"img":"v+zoJ0tHyJb+Zc6Mlpcrwg==","content":"

2001年生。硕士研究生。主要研究方向为推荐系统、大语言模型

"}, bioImg=v+zoJ0tHyJb+Zc6Mlpcrwg==, bioContent=

2001年生。硕士研究生。主要研究方向为推荐系统、大语言模型

, aboutCorrespAuthor=null)}, companyList=null), Author(id=1325490376827600998, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, orderNo=4, firstName=null, middleName=null, lastName=null, nameCn=null, orcid=null, stid=null, country=null, authorPic=null, dead=0, email=null, emailSecond=null, emailThird=null, correspondingAuthor=0, authorType=1, ext={EN=AuthorExt(id=1325490377263808615, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, authorId=1325490376827600998, language=EN, stringName=Yusong Ye, firstName=Yusong, middleName=null, lastName=Ye, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=null, bio={"content":"

born in 2001. Master candidate. His main research interests include data mining and recommendation system

"}, bioImg=null, bioContent=

born in 2001. Master candidate. His main research interests include data mining and recommendation system

, aboutCorrespAuthor=null), CN=AuthorExt(id=1325490377746153576, tenantId=1146029695717560320, journalId=1324759914146529478, articleId=1325490262708977701, authorId=1325490376827600998, language=CN, stringName=叶育松, firstName=null, middleName=null, lastName=null, prefix=null, suffix=null, authorComment=null, nameInitials=null, affiliation=null, department=null, xref=null, address=null, bio={"img":"oLR0vp92nEHoQRjKBHejkw==","content":"

2001年生。硕士研究生。主要研究方向为数据挖掘、推荐系统

"}, bioImg=oLR0vp92nEHoQRjKBHejkw==, bioContent=

2001年生。硕士研究生。主要研究方向为数据挖掘、推荐系统

, aboutCorrespAuthor=null)}, companyList=null)]
李佚名 , 于亚新* , 于之晟, 司一廷, 叶育松
作者信息
  • 东北大学计算机科学与工程学院 沈阳 110169
  • 医学影像智能计算教育部重点实验室(东北大学) 沈阳 110169
通讯作者:
于亚新()
作者简介:

2000年生。硕士研究生。主要研究方向为知识图谱、推荐系统

1971年生。博士,副教授,硕士生导师。IEEE,ACM,CCF会员。主要研究方向为数据挖掘、社交网络

2001年生。硕士研究生。主要研究方向为推荐系统、因果关系推理

2001年生。硕士研究生。主要研究方向为推荐系统、大语言模型

2001年生。硕士研究生。主要研究方向为数据挖掘、推荐系统

Dual-Perspective Multi-Level Cross-Modal Recommendation Based on Large Language Models
Yiming Li , Yaxin Yu* , Zhisheng Yu, Yiting Si, Yusong Ye
Affiliations
  • School of Computer Science and Engineering, Northeastern University, Shenyang 110169
  • Key Laboratory of Intelligent Computing in Medical Image (Northeastern University), Ministry of Education, Shenyang 110169
  • born in 2000. Master candidate. His main research interests include knowledge graph and recommendation system

    born in 1971. PhD, associate professor, master supervisor. Member of IEEE, ACM and CCF. Her main research interests include data mining and social network

    born in 2001. Master candidate. His main research interests include recommendation system and causal relationship inference

    born in 2001. Master candidate. His main research interests include recommendation system and large language model

    born in 2001. Master candidate. His main research interests include data mining and recommendation system

出版时间: 2026-01-01 doi: 10.7544/issn1000-1239.202550040
文章导航
收藏切换

多模态推荐系统旨在提供更为精准和个性化的推荐服务。然而,现有研究仍存在以下问题:1)特征失真。由于输入的嵌入均由小型预训练语言模型和深层卷积神经网络等模型进行处理,导致得到的特征表示不准确。2)编码视角单一。目前模型的多模态编码层只考虑在单一的记忆或扩展视角进行编码,造成信息缺失。3)多模态对齐效果欠佳。不同模态嵌入分布在不同空间中,需将其映射至同一空间以实现对齐。而现有方法通过简单的行为信息乘积无法捕捉模态之间的复杂关系,导致多种模态无法精确对齐。基于上述问题,提出了一个新颖的模型DPRec。该模型同时考虑了记忆与扩展的双视角编码,并引入超图进行多级精准跨模态对齐。所提模型在3个真实数据集上进行了扩展实验,实验结果验证了所提模型的有效性。

多模态推荐  /  特征表示  /  编码视角  /  超图  /  跨模态对齐

Multimodal recommendation systems aim to deliver more accurate and personalized recommendations by leveraging diverse data modalities such as text and images. Despite their potential, existing approaches face several critical challenges. 1) Feature distortion. The input embeddings are processed by relatively small pre-trained language models and deep convolutional neural networks, which may lead to inaccurate and incomplete feature representations. 2) Single encoding perspective. The multimodal encoding layers of current models only consider encoding from a single perspective that is either memory-based or expansion-based, which limits representational capacity and leads to information loss. 3) Poor multimodal alignment. Embeddings from different modalities are distributed in distinct and heterogeneous semantic spaces, and need to be effectively mapped into a shared semantic space to achieve accurate alignment. However, existing methods typically rely on simple operations such as direct product of behavioral information, which cannot adequately capture the complex and high-order relationships among modalities. As a result, precise alignment across multiple modalities is difficult to achieve. To address these issues, we propose a novel model named DPRec. This model considers encoding from both memory and expansion perspectives and introduces hypergraphs for multi-level precise cross-modal alignment. The model is evaluated through extensive experiments on three real-world datasets. Experimental results confirm the effectiveness of DPRec, demonstrating its superiority over state-of-the-art approaches in improving recommendation accuracy.

multimodal recommendation  /  feature representations  /  encoding perspective  /  hypergraph  /  cross-modal alignment
李佚名, 于亚新, 于之晟, 司一廷, 叶育松. 基于大语言模型的双视角多级跨模态推荐. 计算机研究与发展, 2026 , 63 (1) : 147 -161 . DOI: 10.7544/issn1000-1239.202550040
Yiming Li, Yaxin Yu, Zhisheng Yu, Yiting Si, Yusong Ye. Dual-Perspective Multi-Level Cross-Modal Recommendation Based on Large Language Models[J]. Journal of Computer Research and Development, 2026 , 63 (1) : 147 -161 . DOI: 10.7544/issn1000-1239.202550040
推荐系统是基于海量数据挖掘的一种智能平台,它根据用户的个人信息和物品特征(如用户的兴趣、历史购买行为和物品的材质、价格等),利用机器学习等人工智能技术建立模型,从而预测用户对新物品的评价和喜好[1]。然而,在大规模的用户和物品空间中,用户与物品之间的交互数据往往非常稀疏,因此存在数据稀疏、冷启动等问题。此外,传统推荐往往仅依赖于用户的历史交互信息,忽视了其他可能影响用户偏好的丰富信息源。
为了解决数据稀疏和单一信息源等问题,多模态推荐系统被提出,如图1所示。多模态推荐系统可以整合不同模态的信息,例如图片、文本、音频和视频等。通过将多种模态信息进行编码和融合,得到用户和物品的嵌入,从而进行推荐。多模态推荐方法致力于将多模态嵌入与用户行为嵌入进行精确融合,以准确捕捉用户的偏好和物品属性信息。
目前,许多研究已将多模态信息集成到推荐系统中。例如,VBPR[2]和CKE[3]通过整合视觉特征扩展了矩阵分解(MF)[4]方法。图卷积网络(GCN)[5-7]因其能捕获高阶连接并强化偏好特征,正逐渐成为研究热点。例如,MMGCN[8]将不同模态的信息融合到多个用户-项目图中,并通过连接学习到的模态表示来对用户偏好进行建模。GRCN[9]在MMGCN的基础上利用多模态特征进一步细化了用户-项目图,旨在剔除虚假交互。DualGNN[10]构建了一个额外的同构用户-用户图,以探索普遍的用户偏好模式。这些基于 GCN 的方法取得了巨大成功,并达到了最先进的性能水平。
多模态推荐相比于传统推荐,能够更全面地理解和刻画用户需求与物品特征,从而显著提升推荐的准确性和个性化程度。例如,在衣服推荐的场景中,如图2所示,多模态系统能够深入挖掘衣服的图片信息和描述文本,甚至结合用户对衣服的评论反馈,通过融合这些多模态信息,系统可以捕捉到用户对时尚的独特偏好,比如用户既喜欢某款衣服的明亮色彩和时尚剪裁,又注重其舒适材质和适合日常穿搭的实用性。因此,系统能够为用户推荐出既符合其审美又满足实际穿着需求的多样化衣服选项,极大地丰富了用户的购物体验,提高了推荐的满意度和转化率。
现有的多模态推荐方法虽然取得了显著成功,但仍存在3种局限性。1)目前,多模态模型在处理文本和图像特征嵌入时,分别使用小型预训练语言模型和深层CNN等模型处理。例如sentence2Vec[11],VGG16[12]等。由于该类模型参数规模相对较小,它们并不具备强大的自然语言处理能力以及图片处理能力,这可能导致特征失真,进而影响整体推荐性能。2)目前对于多模态编码层的研究[8-10]主要集中于从单一记忆视角进行编码,也有少数研究[13]尝试从扩展视角进行编码。这2种视角的编码分别通过构建用户-项目图及扩展用户-项目图来实现。然而,这种单一视角编码容易导致编码信息的缺失。正确的做法应是将记忆信息作为主要编码信息,同时将扩展信息作为辅助补充,这样才能获得信息丰富的嵌入。换句话说,目前研究得到的多模态编码层信息不充分,导致多模态融合层无法得到精确的用户项目嵌入。3)现有研究通过简单地利用行为信息与不同模态嵌入进行乘积等相关运算,使其映射至与行为信息同一空间之中。但是,简单的向量拼接、元素级相乘等方法缺乏表达能力,无法充分捕捉不同模态之间的复杂关系。
为了解决上述问题,提出了一种融合记忆与扩展的GCN推荐模型DPRec。该模型的输入是经过大语言模型(large language model,LLM)处理的嵌入向量,并构建了一个同时考虑记忆与扩展的双视角信息编码层,以获取更加丰富的模态信息。同时,利用超图进行多级跨模态精准对齐,将多种模态嵌入映射至同一空间中,实现模态对齐。
本文的主要贡献有4个方面:
1)提出了利用LLM进行多模态数据特征表示。同时,从不同模态信息中抽取实体词,并将其与用户行为信息相结合。该方法避免了特征失真,并从更细粒度的层面提升了模型的记忆能力。
2)提出了同时考虑记忆与扩展的双视角信息编码。该模块通过记忆与扩展2个图神经网络,有效抽取了记忆性与扩展性信息,避免了信息缺失。
3)提出了利用超图与双视角编码得到的模态嵌入进行多级跨模态精准对齐的方法。该方法可有效捕捉模态之间的复杂关系,使多种模态嵌入映射至同一空间,从而在模态融合层实现精确融合。
4)在3个公开数据集上进行了扩展实验,实验结果表明,所提模型在性能指标上均优于其他对比模型,验证了本文方法的有效性。
随着模型参数量和训练数据量的不断增加,语言模型的文本建模困惑度以可预测的方式降低,在各类自然语言处理任务上的表现也持续提升[14]。基于BERT[15]和Transformer[16]的大语言模型通过自注意力机制和深度学习架构,极大地提升了文本处理效果,使其在推荐系统领域的应用越来越广泛。例如,LLMRec[17]通过3种简单但有效的基于LLM的图增强策略来增强推荐系统。IDGenRec[18]使用人类语言标记将每个项目表示为一个独特、简洁、语义丰富且平台无关的文本ID,通过训练一个文本ID生成器与基于LLM的生成器协同工作,使得个性化推荐能够无缝集成到自然语言生成中。TALLRec[19]通过推荐数据对LLM进行调整,构建了一个大型推荐语言模型,用于将LLM与推荐任务进行对齐。
以上推荐方法主要是在推荐过程中应用LLM,但并没有从根本上解决特征失真问题。DPRec模型通过使用2种不同的LLM来对多模态数据进行特征表示,从而获得多模态数据的精确嵌入。同时,利用LLM对多模态数据进行实体词抽取,并将这些实体词转换为嵌入向量与用户行为信息相结合。这有助于更好地获得记忆视角的信息,从而提高整体推荐效果。
多模态推荐已成为网络媒体平台的重要应用之一。通过分析海量的多模态信息(如图像、文本)以及用户的历史行为数据,能够为用户提供高度个性化的推荐服务。这一过程不仅提升了用户体验,还能增强平台的用户粘性和满意度。例如,VPBR和VPOI[20]利用预训练的CNN模型来提取深度视觉特征,以此丰富项目表示。ACF[21]通过注意力机制自适应地捕获用户对每种模态的偏好。LATTICE[22]引入了项目语义图,以揭示项目之间的潜在相关信号。FREEDOM[23]在LATTICE基础上,通过降低用户-项目交互图的噪声,并在初始化后固定项目语义图,进一步提升了推荐精度。最近,MKGAT[24]引入了图注意力机制,以平衡模态融合过程中不同模态的权重。然而,现有方法在多模态编码层仍拘泥于使用单一视角的用户-项目图或扩展用户-项目图,并且多模态对齐问题没有得到很好的解决,从而限制了推荐系统的整体表现。
超图作为一种能够捕捉复杂关系的数据结构,在推荐系统中展现出了独特的应用潜力。超图卷积神经网络本质上是将GCN显式或隐式地扩展到超图上,或者引入注意力机制,设计出适用于超图的卷积操作[25]。在推荐系统领域,超图可以用来构建用户与商品之间的多维关系模型,例如,用户对商品的评分、浏览历史、购买行为等,这些都可以作为超边连接多个节点的依据。超图神经网络(HGNN)[26]是图神经网络的扩展,它能够处理超图结构数据,更精准地捕捉用户和商品之间的复杂交互模式。HGNN通过超边卷积操作利用高阶数据相关性进行表示学习,在处理多模态、异构数据时显示出了优势。为了避免构建超图时产生过多虚假关联,通常使用函数对超图进行参数化,例如Gumbel-Softmax[27]。此外,超图对比学习也被提出用于推荐系统,通过构建超图结构并进行对比学习,可以提高推荐系统在处理噪声和倾斜分布数据时的性能。最近的研究还提出了自监督超图Transformer(self-supervised hypergraph Transformer, SHT)[28],这是一种新框架,它通过探索全局协作关系来增强用户嵌入,并使用跨视图生成自监督学习组件进行数据增强,以增强推荐系统的鲁棒性。
DPRec利用超图学习到的模态嵌入与双视角编码得到的模态嵌入进行多级跨模态精准对齐,实现了将多种模态嵌入映射到同一空间的目标,从而提高模型的整体性能。
在本节中,将详细描述所提出的模型DPRec。表1中记录了本文出现的主要符号的描述。
定义1. 记忆视角。记忆视角指根据用户集$ U $与项目集$ I $的交互矩阵$ \boldsymbol{R}\in {\left\{\mathrm{0,1}\right\}}^{\left|U\right|\times \left|I\right|} $,构建对应的二分图$ G $。而后将$ {\boldsymbol{E}}_{i,{\mathrm{id}}} $,$ {\boldsymbol{E}}_{\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{t}\mathrm{y}} $,$ {\boldsymbol{E}}_{i,m} $等输入图$ G $中进行图卷积操作得到$ {\boldsymbol{E}}_{m\_\mathrm{r}\mathrm{e}\mathrm{p}} $,其中$ m\in \left\{{\mathrm{id}},{\mathrm{t}},{\mathrm{v}}\right\} $。记忆视角编码$ Memery $表示为$ \displaystyle\sum\limits_{m \in \{ {\mathrm{id}},{\mathrm{t}},{\mathrm{v}}\} } {\mathop E\nolimits_{m\_{\text{rep}}} } $。
定义2. 扩展视角。扩展视角指在二分图$ G $基础上,利用项目$ I $之间的相似度对$ G $进行扩展,得到扩展图$ \tilde{G} $。将$ {\boldsymbol{E}}_{u,{\mathrm{id}}}^{ \wedge } $,$ {\boldsymbol{E}}_{i,{\mathrm{id}}}^{ \wedge } $,$ {\boldsymbol{E}}_{i,{\mathrm{v}}}^{+} $,$ {\boldsymbol{E}}_{i,{\mathrm{t}}}^{+} $,$ {\boldsymbol{E}}_{u,{\mathrm{v}}} $,$ {\boldsymbol{E}}_{u,{\mathrm{t}}} $等输入$ \tilde{G} $中进行图卷操作得到$ {\boldsymbol{E}}_{m}^{\sim } $,其中$ m\in \left\{{\mathrm{id}},{\mathrm{t}},{\mathrm{v}}\right\} $。扩展视角编码$ Expand $表示为$\displaystyle\sum\limits_{m \in \{ {\mathrm{id}},{\mathrm{t}},{\mathrm{v}}\} } {\mathop E\nolimits_{m\_{\rm{rep}}} } $。
定义3. 双视角。双视角指在多模态编码层搭建用户-项目图$ G $与扩展图$ \tilde{G} $,同时考虑在记忆与扩展这2种视角进行信息编码。双视角(DP)编码可以表示为$ Memery\cup Expand $,其中$ \cup $表示将二者沿着维度为1进行拼接。
定义4. 多级跨模态对齐。多级跨模态对齐是指在3个不同级别上进行模态之间的对齐。划分为3个级别进行模态对齐,旨在从更细粒度的层面提高模态对齐效果。3个级别分别表示为
$ {L}_{1}={\boldsymbol{E}}^{{\mathrm{t}},{H}} \leftrightarrow {\boldsymbol{E}}_{{\mathrm{v}}\_\mathrm{r}\mathrm{e}\mathrm{p}} , $
$ {L}_{2}={\boldsymbol{E}}^{{\mathrm{v}},{H}} \leftrightarrow {\boldsymbol{E}}_{{\mathrm{t}}\_\mathrm{r}\mathrm{e}\mathrm{p}} , $
$ L_3=\boldsymbol{E}^H\leftrightarrow\boldsymbol{E}_{\mathrm{id}_{\mathrm{rep}}}+\boldsymbol{E}^H\leftrightarrow\tilde{\boldsymbol{E}}_{\mathrm{t}}+\boldsymbol{E}^H\leftrightarrow\tilde{\boldsymbol{E}}_{\mathrm{v}}\; , $
其中$ {L}_{1} $,$ {L}_{2} $,$ {L}_{3} $表示多级别对齐产生的对齐损失,$ \leftrightarrow $表示在不同嵌入之间参数化计算模态之间的距离,$H $表示图卷积总层数。
根据定义1~4可大致描述本文要解决的问题及简要过程。在多模态编码层只考虑记忆视角得到$ Memery $或只考虑扩展视角得到$ Expand $均会导致编码信息不充分。基于此,提出在编码层构建一个双视角编码由$ Memery\cup Expand $得到,既考虑了记忆信息也考虑了扩展信息。除此之外,由于不同模态之间分布差异巨大,不可直接进行多模态融合。基于此,提出利用超图学习到的丰富信息与双视角编码层得到的信息进行多级跨模态对齐操作,即上述${\boldsymbol{E}}^{{\mathrm{t}},{H}} \leftrightarrow $$ {\boldsymbol{E}}_{{\mathrm{v}}\_\mathrm{r}\mathrm{e}\mathrm{p}} $,$ {\boldsymbol{E}}^{{\mathrm{v}},{H}} \leftrightarrow {\boldsymbol{E}}_{{\mathrm{t}}\_\mathrm{r}\mathrm{e}\mathrm{p}} $,$ \boldsymbol{E}^H\leftrightarrow\boldsymbol{E}_{\mathrm{id}\_\mathrm{rep}}+\boldsymbol{E}^H\leftrightarrow\tilde{\boldsymbol{E}}_{\mathrm{t}}+\boldsymbol{E}^{\mathrm{\mathit{H}}}\leftrightarrow\tilde{\boldsymbol{E}}_{\mathrm{v}} $。经上述操作之后,将得到同时富含记忆与扩展且分布在同一空间中的多种模态信息。便于之后的模态融合得到准确的用户项目嵌入,提高推荐效果。
模型的总体架构如图3所示,其中主要包含2个模块:
1)多模态双视角信息编码层。在保留传统用户-项目图的同时引入扩展用户-项目图,旨在同时抽取记忆性与扩展性信息,得到富含双视角信息的用户项目嵌入,避免信息缺失,如图4所示。
2)多模态对齐层。利用HGNN学习到的不同模态嵌入、混合模态嵌入与双视角编码得到的不同模态嵌入进行3种级别的跨模态精准对齐,如图5所示。
在本研究中,摒弃了之前使用小型预训练语言模型和CNN进行数据处理的方法,转而利用LLM将多模态数据处理为嵌入表示,以发挥其本身的自然语言处理能力和图片处理方面的优势。具体来说,使用了2种预训练LLM来分别处理文本和图像数据。此外,还利用LLM执行其擅长的实体词抽取任务。在抽取实体词方面,通过设计合适的提示词模板从文本和图像中抽取关键实体词。对于文本数据,设计了如下提示词:“从该段文本中抽取出你认为所有可能的实体词”。对于图像数据,使用了描述性的提示词,如“请提取出该图片中的实体词,只需提取出关键主题词,一些微小因素可进行适当忽略。例如,如果图片中有一个人,描述他们的服装、姿势和表情等相关词语”。通过这些提示词模板,LLM能够自动提取出品牌名、产品类型等关键实体。同时,将抽取出来的实体词转换为嵌入向量,并将其与用户行为信息相结合,旨在更细粒度地捕捉用户兴趣偏好,丰富项目属性信息,从而增强推荐系统的准确性。
现有的多模态方法面临着难以精确获取记忆信息和扩展信息的挑战。鉴于此,本文创新性地提出了一种同时考虑记忆与扩展的双视角多模态信息编码器,旨在获取更丰富的多模态用户项目嵌入,以解决现有方法中嵌入信息缺失的问题。
在基于GCN的推荐系统中,盲目引入高阶节点可能会导致虚假信息在传播过程中被引入。受GUME[13]的启发,应采取一种更为精细化的方法:仅为每个项目保留相似度得分最高的前$ K $个邻居来构建扩展用户-项目图。这一方法有助于识别出在多种模式下与目标项目相似的项目,但GUME仅将扩展图得到的用户与项目模态嵌入进行串联拼接并未将行为扩展信息引入其中。本文的扩展视角在得到扩展的模态与行为嵌入后,利用行为嵌入与模态嵌入相乘,引入行为信息。
首先,采用KNN算法为每种模态$ m $构建一个项目-项目图,目的在于挖掘项目之间的多模态关系。之后,通过计算项目对$ \left(i,j\right)\in I $的原始模态特征$ {\boldsymbol{e}}_{i}^{m} $和$ {\boldsymbol{e}}_{j}^{m} $的余弦相似度来计算相似度得分$ {S}_{i,j}^{m} $。
$ {S}_{i,j}^{m}=\frac{{\left({\boldsymbol{e}}_{i}^{m}\right)}^{\mathrm{T}}{\boldsymbol{e}}_{j}^{m}}{\left|\right|{\boldsymbol{e}}_{i}^{m}\left|\right|\mathrm{ }\mathrm{ }\mathrm{ }\mathrm{ }\mathrm{ }\mathrm{ }\left|\right|{\boldsymbol{e}}_{j}^{m}\left|\right|} \;。 $
只保留相似度得分最高的前$ K $个邻居来捕获最相关的特征。
$ {\bar{S}}_{i,j}^{m}=\left\{\begin{aligned}&{S}_{i,j}^{m},\quad{S}_{i,j}^{m}\in topK\left(\left\{{S}_{i,p}^{m},p \in I\right\}\right),\\&0,\quad 其他,\end{aligned}\right.$
其中$ {\bar{S}}_{i,j}^{m} $表示模态$ m $内项目$ i $和项目$ j $之间的边权重,$ \{{S}_{i,p}^{m},p\in I\} $表示第$ i $个项目的邻居分数。同时,为了减缓梯度爆炸或者消失问题,将邻接矩阵标准化为:
$ {\tilde{\boldsymbol{S}}}^{m}={\left({\boldsymbol{D}}^{m}\right)}^{-\frac{1}{2}}{\bar{\boldsymbol{S}}}^{m}{\left({\boldsymbol{D}}^{m}\right)}^{-\frac{1}{2}} \text{,} $
其中$ {\boldsymbol{D}}^{m} $是$ {\bar{\boldsymbol{S}}}^{m} $的对角线矩阵,在初始化后便固定每个模态项图。
然后,将这些项目定义为目标项目的语义邻居。项目集及其语义邻居可以表示为
$ C = \bigcup\limits_{i \in I} {\{ (i,j)|j \in \mathop {\boldsymbol{N}}\nolimits_i \} } \text{,} $
其中$ {N}_{i} $表示项目$ i $的语义邻居。然后,通过在项目和它们的语义邻居之间添加边来扩展用户-项目图。形式上:
$ \boldsymbol{\gamma }=\left(\begin{array}{cc}{\boldsymbol{0}}& \boldsymbol{R}\\ {\boldsymbol{R}}^{\mathrm{T}}& \boldsymbol{C}\end{array}\right) \text{,} $
其中0是全0矩阵,$ \boldsymbol{R} $是用户-项目交互矩阵。
扩展图成功构建后,对项目同质图执行图卷积操作以传播和更新模态特征:
$ \boldsymbol{E}_{i,m}^+=\tilde{\boldsymbol{S}}^m\boldsymbol{E}_{i,m}\text{。} $
同时,在扩展用户-项目图上传播特定模态的嵌入,以提取特定于模态的扩展兴趣嵌入,这同样适用于ID嵌入。具体来说,在图卷积的第$ l $层,用户和项目嵌入可以表示为
$ {\boldsymbol{E}}_{m}^{ \sim \left(l\right)}=\mathrm{ }\left({\boldsymbol{D}}^{-\frac{1}{2}}\boldsymbol{\gamma }{\boldsymbol{D}}^{-\frac{1}{2}}\right){\boldsymbol{E}}_{m}^{ \sim (l-1)} \text{,} $
其中$ {\boldsymbol{E}}_{m}^{ \sim \left(l\right)} $表示图卷积第$ l $层用户和项目的特定模态嵌入, $ \boldsymbol{D} $是$ \boldsymbol{\gamma } $的对角度矩阵,$ {\boldsymbol{E}}_{m}^{\sim \left(0\right)} $是${\boldsymbol{E}}_{u,m} $和$ {\boldsymbol{E}}_{i,m}^{+} $的串联。
通过聚合多层邻居信息,可以获得最终特定模态的扩展兴趣特征:
$ \mathop {\boldsymbol{E}}\nolimits_m^{ \sim }= \frac{1}{{{{L}} + 1}}\sum\limits_{i = 0}^{{L}} {\mathop {\boldsymbol{E}}\nolimits_m^{ \sim (l)} } \text{,} $
其中$ {L} $是用户-项目图层数。
最后,利用扩展视角得到的用户行为信息与其余模态信息进行相乘,目的是进行特征增强,将扩展行为信息引入其余扩展模态嵌入中。
为了捕获记忆视角信息,首先为每个模态分别构建一个用户-项目图$ G $。之后对$ {\boldsymbol{E}}_{i,{\mathrm{id}}} $进行更新,利用LLM抽取的实体词使其用户行为信息更加丰富。
$ {\boldsymbol{E}}_{i,{\mathrm{id}}}={\boldsymbol{E}}_{i,{\mathrm{id}}}+{\boldsymbol{E}}_{\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{t}\mathrm{y}} \text{,} $
其中$ {\boldsymbol{E}}_{\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{t}\mathrm{y}} $是由LLM进行实体词抽取后转化得到的嵌入向量。
在第$ l $个图卷积层上用户和项目表示可以公式化为
$ {\boldsymbol{E}}_m^{(l)} = \sum\limits_{i \in N_u } {\frac{1}{{\sqrt {\left| {{{N}}_u } \right|\left| { {{N}}_i } \right|} }}} {\boldsymbol{E}}_m^{(l - 1)} , $
其中$ {N}_{u} $和$ {N}_{i} $分别代表图$ G $中$ u $和$ i $的1跳邻居,$ {\boldsymbol{E}}_{m} $为$ {\boldsymbol{E}}_{i,m} $和记忆视角用户嵌入串联获得,记忆视角用户嵌入由聚合邻居项目嵌入得到。
每种模态的最终嵌入都是通过逐元素求和来计算。形式上:
$ {\boldsymbol{E}}_{m\_{\text{rep}}} = \sum\limits_{l=0}^{{L}} { {\boldsymbol{E}}_m^{(l)} } , $
其中$ {L} $是用户-项目图层数。
然后将$ {\boldsymbol{E}}_{i,m} $放进$ G $中进行图卷积操作,利用式(13)(14)进行计算操作,最终可以得到$ {\boldsymbol{E}}_{m\_\mathrm{r}\mathrm{e}\mathrm{p}} $。其中$ m\in \left\{{\mathrm{id}},{\mathrm{t}},{\mathrm{v}}\right\} $。
本节利用超图得到的丰富模态嵌入与双视角编码得到的嵌入信息进行多级跨模态精准对齐,目的是将不同模态的嵌入映射到同一空间中。
LGMRec[29]将超图学习到的文本与图像模态嵌入利用对比学习方法拉近二者之间距离,并将二者相加,与该模型记忆视角得到的嵌入相结合得到最终的嵌入。本文受其超图应用的启发,在保留消息传递得到2种模态嵌入之外,将二者相加得到超图混合模态嵌入,并弃用利用对比学习直接拉近其距离的方式。
首先,本文将可学习的隐式属性向量$ {{\boldsymbol{v}}_{a}^{m}}\in {\mathbb{R}}^{d_m},a= 1, 2,…,N $ 定义为模态$ m $的超边,自适应地学习隐式属性以及项目与用户之间的依赖关系,其中$ N $是超边数量。具体来说,通过以下方式获得低维嵌入空间中的超图依赖矩阵:
$ {\boldsymbol{H}}_{i}^{m}={\boldsymbol{E}}_{i,m}\cdot (({\boldsymbol{V}}^{{m}})^{\mathrm{T}}),{\boldsymbol{H}}_{u}^{m}={\boldsymbol{A}}_{u}\cdot (({{\boldsymbol{H}}_{i}^{m}})^{\mathrm{T}}) , $
其中$ {\boldsymbol{H}}_{i}^{m}\in {\mathbb{R}}^{\left|I\right|\times N} $和$ {\boldsymbol{H}}_{u}^{m}\in {\mathbb{R}}^{\left|U\right|\times N} $是项目和用户的超边依赖矩阵,$ {\boldsymbol{V}}^{m}=\left({\boldsymbol{V}}_{1}^{m},{\boldsymbol{V}}_{2}^{m},… ,{\boldsymbol{V}}_{N}^{m}\right)\in {\mathbb{R}}^{N\times {d}_{m}} $为超边向量矩阵,$ {\boldsymbol{A}}_{u}\in {\mathbb{R}}^{\left|U\right|\times \left|I\right|} $ 是从A中提取的与用户相关的邻接矩阵。
为避免构建超图时产生虚假关联,采用Gumbel-Softmax重新进行参数化,以确保每个项目尽可能仅附加到1个超边:
$ {\tilde{\boldsymbol{h}}}_{i,*}^{m}=softmax\left(\dfrac{\mathrm{ln}\delta -\mathrm{ln}\left(1-\delta \right)+{\boldsymbol{h}}_{i,*}^{m}}{\tau }\right) \text{,} $
其中$ {\boldsymbol{h}}_{i,*}^{m}\in {\mathbb{R}}^{N} $是项目超边依赖矩阵中第$ i $行向量,反映了项目$ i $与所有超边之间的关系。$ \boldsymbol{\delta }\in {\mathbb{R}}^{N} $是噪声向量,其中,每个值$ {\delta }_{j}\sim Uniform\left(\mathrm{0,1}\right) $, $ \tau $是温度超参数。经过以上操作,可以得到增强项目属性超图依赖矩阵$ {\tilde{\boldsymbol{H}}}_{i}^{m}。$同理,也可以得到增强用户属性超图依赖矩阵$ {\tilde{\boldsymbol{H}}}_{u}^{m} $。
通过将属性超边作为中间枢纽,可以实现超图消息传递,将全局信息传递给用户和项目,而不受几跳距离的限制。形式上:
$ {\boldsymbol{E}}_{i}^{m,h+1}=drop({\tilde{\boldsymbol{H}}}_{i}^{m})\cdot drop\left(({\tilde{\boldsymbol{H}}}_{i}^{{m}})^{\mathrm{T}}\right) \text{,} $
其中$ {\boldsymbol{E}}_{i}^{m,h} $是第$ h $超图层中项目全局嵌入矩阵,drop代表dropout函数。使用记忆视角得到的项目协作嵌入$ {\boldsymbol{E}}_{{\mathrm{id}}\_\mathrm{r}\mathrm{e}\mathrm{p}} $作为$ h=0 $时的全局嵌入矩阵初始化。类似地,全局用户嵌入矩阵可以计算为
$ {\boldsymbol{E}}_{u}^{m,h+1}=drop({\tilde{\boldsymbol{H}}}_{u}^{m})\cdot drop\left(({\tilde{\boldsymbol{H}}}_{u}^{{m}})^{\mathrm{T}}\right) \;。 $
然后,可以通过聚合所有模态的全局嵌入来获得全局嵌入矩阵EH,
$ {\boldsymbol{E}}^{{H}} = \alpha \cdot norm\left(\sum\limits_{m \in M} {{\boldsymbol{E}}^{m,{{H}}} } \right)\text{,} $
$ {\boldsymbol{E}}^{m,{H}}=\left({\boldsymbol{E}}_{u}^{m,{H}},{\boldsymbol{E}}_{i}^{m,{H}}\right) , $
其中$ {\boldsymbol{E}}_{u}^{m,{H}}\in {\mathbb{R}}^{\left|U\right|\times d} $,$ {\boldsymbol{E}}_{i}^{m,{H}}\in {\mathbb{R}}^{\left|I\right|\times d} $分别是在模态$ m $下第$ {H} $超图层中获得的用户$ u $与项目$ i $的全局嵌入矩阵,$ {H} $为超图卷积总层数,norm是一个标准化函数,用于减轻嵌入之间的值尺度差异,$ \alpha $是一个可调节因子,用于控制全局嵌入的结合程度。
模态对齐一直是热点讨论问题。MENTOR[30]中提出了一个4级跨模态对齐方式,包括行为直接指导、行为间接指导、模态直接对齐和模态间接对齐,旨在将记忆视角得到的多种模态嵌入实现多种不同程度的对齐以拉近不同模态之间的距离。受其启发,本文利用超图得到的3种模态嵌入与双视角得到的嵌入来实现跨模态对齐,对齐分为3个级别:1)超图文本嵌入与记忆视角图像嵌入对齐;2)超图图像嵌入与记忆视角文本嵌入对齐;3)超图混合模态嵌入与记忆视角行为嵌入以及扩展视角文本与图像嵌入均进行对齐。为高效实现跨模态对齐,需采用一种方法来量化和比较不同模态之间的相似性和差异性。高斯分布参数化是一种自然且有效的选择,其由均值和方差2个参数完全确定。通过参数化模态嵌入为高斯分布,可直观地衡量不同模态之间的差异。此外,分布之间的相似性可通过KL散度来量化,值越小表示2个分布越接近,这为多级对齐提供了理论基础。假设有2个嵌入分别为$ {\boldsymbol{E}}_{1} $,$ {\boldsymbol{E}}_{2} $,将其参数化为$ {\boldsymbol{E}}_{1} \sim N\left({\mu }_{1},{\sigma }_{1}^{2}\right) $,$ {\boldsymbol{E}}_{2} \sim N\left({\mu }_{2},{\sigma }_{2}^{2}\right) $,二者之间的KL散度为
$ {D}_{\mathrm{K}\mathrm{L}}=\mathrm{ln}\left(\frac{{\sigma }_{2}}{{\sigma }_{1}}\right)+\frac{{\sigma }_{1}^{2}+{({\mu }_{1}-{\mu }_{2})}^{2}}{{2\sigma }_{2}^{2}}-\frac{1}{2} , $
可等效简化为
$ {D}_{\mathrm{K}\mathrm{L}}\leftrightarrow \left|{\mu }_{1}-{\mu }_{2}\right|+|{\sigma }_{1}-{\sigma }_{2}| \;。 $
由此可以得到所提3级对齐的表示方式。
1)第1级别为:
$ {\boldsymbol{E}}^{{\mathrm{t}},{H}}\sim N\left({\mu }_{{\mathrm{ct}}},{\sigma }_{{\mathrm{ct}}}^{2}\right),{\boldsymbol{E}}_{{\mathrm{v}}\_\mathrm{r}\mathrm{e}\mathrm{p}}\sim N\left({\mu }_{{\mathrm{v}}},{\sigma }_{{\mathrm{v}}}^{2}\right) , $
$ {L}_{\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{g}\mathrm{n}1}=\left|{\mu }_{{\mathrm{ct}}}-{\mu }_{{\mathrm{v}}}\right|+\left|{\sigma }_{{\mathrm{ct}}}-{\sigma }_{{\mathrm{v}}}\right| \text{,} $
其中$ {(\mu }_{{\mathrm{ct}}},{\sigma }_{{\mathrm{ct}}}) $与$ ({\mu }_{{\mathrm{v}}},{\sigma }_{{\mathrm{v}}}) $分别表示超图文本嵌入与记忆视角图像嵌入的高斯分布。
2)第2个级别为:
$ {\boldsymbol{E}}^{{\mathrm{v}},{H}}\sim N\left({\mu }_{{\mathrm{cv}}},{\sigma }_{{\mathrm{cv}}}^{2}\right),{\boldsymbol{E}}_{{\mathrm{t}}\_\mathrm{r}\mathrm{e}\mathrm{p}}\sim N\left({\mu }_{{\mathrm{t}}},{\sigma }_{{\mathrm{t}}}^{2}\right) \text{,} $
$ {L}_{\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{g}\mathrm{n}2}=\left|{\mu }_{{\mathrm{cv}}}-{\mu }_{{\mathrm{t}}}\right|+\left|{\sigma }_{{\mathrm{cv}}}-{\sigma }_{{\mathrm{t}}}\right| \text{,} $
其中$ {(\mu }_{{\mathrm{cv}}},{\sigma }_{{\mathrm{cv}}}) $与$ ({\mu }_{{\mathrm{t}}},{\sigma }_{{\mathrm{t}}}) $分别表示超图图像嵌入与记忆视角文本嵌入的高斯分布。
3)第3级别为:
$ {\boldsymbol{E}}^{{H}}\sim N\left({\mu }_{{\mathrm{ch}}},{\sigma }_{{\mathrm{ch}}}^{2}\right),{\tilde{\boldsymbol{E}}}_{{\mathrm{t}}}\sim N\left({\mu }_{{\mathrm{et}}},{\sigma }_{{\mathrm{et}}}^{2}\right) \text{,} $
$ {\boldsymbol{E}}_{{\mathrm{id}}\_\mathrm{r}\mathrm{e}\mathrm{p}}\sim N\left({\mu }_{{\mathrm{id}}},{\sigma }_{{\mathrm{id}}}^{2}\right),{\tilde{\boldsymbol{E}}}_{{\mathrm{v}}}\sim N\left({\mu }_{{\mathrm{ev}}},{\sigma }_{{\mathrm{ev}}}^{2}\right) \text{,} $
$\begin{split}{L}_{\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{g}\mathrm{n}3}=\;&\left|{\mu }_{{\mathrm{ch}}}-{\mu }_{{\mathrm{et}}}\right|+\left|{\sigma }_{{\mathrm{ch}}}-{\sigma }_{{\mathrm{et}}}\right|+\left|{\mu }_{{\mathrm{ch}}}-{\mu }_{{\mathrm{id}}}\right|+\\&\left|{\sigma }_{{\mathrm{ch}}}-{\sigma }_{{\mathrm{id}}}\right|+\left|{\mu }_{{\mathrm{ch}}}-{\mu }_{{\mathrm{ev}}}\right|+\left|{\sigma }_{{\mathrm{ch}}}-{\sigma }_{{\mathrm{ev}}}\right| \text{,} \end{split}$
其中$ ({\mu }_{{\mathrm{ch}}},{\sigma }_{{\mathrm{ch}}}) $,$ ({\mu }_{{\mathrm{et}}},{\sigma }_{{\mathrm{et}}}) $,$ ({\mu }_{{\mathrm{id}}},{\sigma }_{{\mathrm{id}}}) $,$ ({\mu }_{{\mathrm{ev}}},{\sigma }_{{\mathrm{ev}}}) $分别表示超图混合模态嵌入、扩展文本嵌入、记忆视角行为信息嵌入、扩展视角图像嵌入的高斯分布。
最终计算总体多级跨模态对齐损失$ {L}_{\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{g}\mathrm{n}} $。形式上:
$ {L}_{\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{g}\mathrm{n}}={\lambda }_{\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{g}\mathrm{n}}\left({L}_{\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{g}\mathrm{n}1}+{L}_{\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{g}\mathrm{n}2}+{L}_{\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{g}\mathrm{n}3}\right) \text{,} $
其中$ {\lambda }_{\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{g}\mathrm{n}} $是平衡超参数。
首先,基于每种模态项目-项目图的最终嵌入来增强记忆视角的项目模态嵌入,表示为:
$ {\boldsymbol{E}}_{m\_\mathrm{r}\mathrm{e}\mathrm{p}}=\left\{{\boldsymbol{E}}_{{\mathrm{v}},m\_\mathrm{r}\mathrm{e}\mathrm{p}}||{\boldsymbol{E}}_{i,m\_\mathrm{r}\mathrm{e}\mathrm{p}}+{\boldsymbol{E}}_{i,m}^+\right\} , $
其中$ \left|\right| $表示串联操作。
然后,以不同的注意力水平融合记忆视角的ID、图像、文本模态。
$ {\boldsymbol{E}}_{\mathrm{r}\mathrm{e}\mathrm{p}}= \{\eta \times {\boldsymbol{E}}_{{\mathrm{v}}\_\mathrm{r}\mathrm{e}\mathrm{p}} \left|\right|\beta \times {\boldsymbol{E}}_{{\mathrm{t}}\_\mathrm{r}\mathrm{e}\mathrm{p}} \left|\right| \kappa \times {\boldsymbol{E}}_{{\mathrm{id}}\_\mathrm{r}\mathrm{e}\mathrm{p}}\} , $
其中$ \eta $,$ \beta $,$ \kappa $是可训练参数,且$ \eta +\beta +\kappa =1 $。
同理,对于扩展视角的模态嵌入,也使用不同的注意力水平融合图像、文本模态:
$ {\boldsymbol{E}}_{\mathrm{e}\mathrm{x}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{d}\_\mathrm{r}\mathrm{e}\mathrm{p}}=\{\varpi \times {\tilde{\boldsymbol{E}}}_{v} \left|\right| \rho \times {\tilde{\boldsymbol{E}}}_{t}\} , $
其中$ \varpi $,$ \rho $是可训练参数,且$ \varpi +\rho =1 $。
最后,将上述得到的记忆视角模态嵌入与扩展视角模态嵌入进行横向拼接,得到富含双视角信息的用户项目嵌入。
$ {\boldsymbol{E}}_{\mathrm{f}\mathrm{i}\mathrm{n}\mathrm{a}\mathrm{l}\_\mathrm{r}\mathrm{e}\mathrm{p}}=\left\{{\boldsymbol{E}}_{\mathrm{r}\mathrm{e}\mathrm{p}} \oplus {\boldsymbol{E}}_{\mathrm{e}\mathrm{x}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{d}\_\mathrm{r}\mathrm{e}\mathrm{p}}\right\} , $
其中$ \oplus $表示横向拼接。
为了优化模型的推荐效果,采用贝叶斯个性化排名(BPR)损失作为主要的优化函数。BPR的目标是扩大每个三元组$ \left(u,p,n\right)\in D $的正样本和负样本之间的预测偏好差距,其中$ D $表示训练集。正样本$ p $是指用户$ u $交互过的项目,而负样本$ n $是从用户$ u $从未互动过的物品集合中随机选择的。BPR函数定义为:
$ \mathop L\nolimits_{{\text{bpr}}} = \sum\limits_{(u,p,n) \in D} { - \ln (\sigma (\mathop y\nolimits_{u,p} - \mathop y\nolimits_{u,n} ))} \text{,} $
其中$ {y}_{u,p} $和$ {y}_{u,n} $是用户$ u $对正样本$ p $和负样本$ n $的评分,由 $ {{\boldsymbol{E}}_{u\_\mathrm{f}\mathrm{i}\mathrm{n}\mathrm{a}\mathrm{l}\_\mathrm{r}\mathrm{e}\mathrm{p}}}^{\mathrm{T}}{\boldsymbol{E}}_{p\_\mathrm{f}\mathrm{i}\mathrm{n}\mathrm{a}\mathrm{l}\_\mathrm{r}\mathrm{e}\mathrm{p}} $ 和 $ {{\boldsymbol{E}}_{u\_\mathrm{f}\mathrm{i}\mathrm{n}\mathrm{a}\mathrm{l}\_\mathrm{r}\mathrm{e}\mathrm{p}}}^{\mathrm{T}}{\boldsymbol{E}}_{n\_\mathrm{f}\mathrm{i}\mathrm{n}\mathrm{a}\mathrm{l}\_\mathrm{r}\mathrm{e}\mathrm{p}} $ 得到。$ \sigma $ 是Sigmoid函数。
然后,结合BPR损失和多级跨模态对齐损失来更新用户和项目表示。如下所示:
$ {L}_{\mathrm{t}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}}={L}_{\mathrm{b}\mathrm{p}\mathrm{r}}+{L}_{\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{g}\mathrm{n}}+\lambda {\left|\right|\varTheta \left|\right|}_{2}^{2} \text{,} $
其中$ \lambda $是控制$ {L}_{2} $正则化的超参数,$ \varTheta $是模型参数集。
在本节中,使用全面的实验来评估DPRec模型在3个广泛使用的现实数据集上的性能,实验中主要验证4个研究问题:RQ1,RQ2,RQ3,RQ4。RQ1:与当前先进的多模态推荐方法相比,本文模型性能是否最优。RQ2:本文模型的各个组件对模型的性能有什么影响。RQ3:本文模型的参数对性能的影响。RQ4:本文模型具体计算开销。
为了评估本文提出的DPRec模型在top-k项目推荐任务中的性能,在亚马逊数据集[31]的3个类别Baby,Sports,Clothing上进行了实验。这些数据集同时提供了产品的描述和图像,本文利用2个预训练好的LLM对数据集进行处理,分别提取512维视觉特征和1536维文本特征。这些数据集的统计数据如表2所示。
为了公平地评估性能,本文采用了2个广泛使用的指标:Recall@k和NDCG@k。在训练、验证和测试阶段,数据集采用8:1:1的随机数据分割。最后,通过报告测试集中所有用户在top-k条件下的平均指标来评估不同方法的推荐性能,其中k值根据经验设置为10和20。
为了证明DPRec模型的有效性,将其与以下最先进的方法进行比较,这些方法可以分为传统推荐方法和多模态推荐方法2组。
1)传统的推荐方法
① MF-BPR[32]。该方法利用BPR损失函数,有效提升了基于矩阵分解的推荐系统性能。
② LightGCN[33]。该方法基于GCN去除了冗余组件,从而简化了模型结构。
③ NCL[34]。该模型是一个图协同过滤框架,通过结合结构邻居和语义邻居来改进神经图协同过滤。
2)多模态推荐方法
① VBPR[2]。该方法通过集成每个项目的视觉和ID嵌入作为其表示来增强MF-BPR的方法,并将其输入到矩阵分解框架中。为了确保评估过程的公正性,进一步将VBPR模型扩展到了文本领域。
② MMGCN[8]。该方法利用GCN为每种模态构建特定的图。最终的预测结果是通过融合所有模态来计算。
③ LATTICE[22]。该方法构建了一个额外的项目语义图来捕获潜在的语义相关信号。
④ MMGCL[35]。该模型通过图对比学习来增强微视频推荐中的多模态嵌入学习。
⑤ SLMRec[36]。该模型将SSL(自监督学习)融入基于图神经网络(GNN)的推荐模型中,并提出了3种数据增强方法,旨在揭示多模态信息中的潜在模式。
⑥ BM3[37]。该模型通过采用潜在表示丢失机制代替图增强来生成对比视图,简化了自监督多模态推荐模型。
⑦ DiffMM[38]。该模型将模态感知图扩散模型与跨模态对比学习相结合,更好地对齐多模态特征信息与协作关系建模。
⑧ MENTOR[30]。该模型提出了一个多层次的自监督学习任务,以解决标签稀疏问题和模态对齐问题。
⑨ LGMRec[29]。该模型提出了利用超图捕获全局用户项目嵌入,同时将得到的全局嵌入与用户-项目图得到的局部嵌入结合,以提高准确性和鲁棒性。
⑩ IDGenRec[18]。该模型通过人类语言标记将每个项目表示为一个独特、简洁、语义丰富且与平台无关的文本ID,从而能够将个性化推荐无缝集成到自然语言生成中。
DPRec以及对比方法均基于PyTorch实现。对于一般设置,本文使用维度为64的Xavier[39] 初始化嵌入,并使用Adam[40]作为优化器。为了实现公平评估,按照已发表的论文对每个基线方法进行完整的网格搜索,以找到最佳设置。对于提出的DPRec,平衡超参数$ {\lambda }_{\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{g}\mathrm{n}} \in\{0.1,0.2,0.3\}$、超边数量$ N\in \{1,2, 4, 8,16,32,64,128,256\} $、可调节因子$ \alpha\in \{0.1,0.4, 0.7, 1.0\} $及邻居数量$ k\in \{5,10,15,20,25,30\} $中进行了网格搜索。学习率固定为1E−4,异构图中的GCN层数设为$ L=2 $。由于考虑收敛,早停阈值固定为20。同时,利用验证集上的Recall@20来作为指标更新最佳纪录。
本文在3个数据集上验证了模型性能,表3显示了DPRec和其他基线在3个数据集上的性能比较。表3揭示了以下观察结果:DPRec模型在多个指标上取得了优异性能,超越了传统推荐模型和多模态推荐模型。具体来说,在Baby,Clothing数据集的Recall@20方面,DPRec分别比最佳基线高出3.1个百分点、0.6个百分点。就NDCG@20而言,分别显示出1.1个百分点、1.2个百分点的提高。就Recall@10而言,分别显示出3.1个百分点、1.1个百分点的改进。最后,对于NDCG@10而言,显示出2.3个百分点、0.6个百分点的提高。对于Sports数据集,DPRec的表现与目前最优模型接近。以上结果验证了DPRec的有效性。
从表3中可以观察到,传统推荐方法的效果远不及DPRec,其原因主要是:1)DPRec整合了多种类型的数据,比单一数据源的传统推荐更丰富。2)DPRec等多模态方法通过不同模态理解用户的意图和物品特征,提高了匹配准确性。3)DPRec可以更好地解决新用户和新物品的推荐问题。4)DPRec的不同模态信息可以相互补充,提高推荐系统的准确性。
除了传统推荐方法之外,从表3还可看出,DPRec与表中的多模态推荐方法相比也表现出更好的性能。虽然它们均是基于GCN的多模态推荐方法,但其他方法大都存在些许不足。例如,LATTICE根据多模态特征相似性动态学习项目之间的潜在结构。然而,动态生成潜在结构是不必要的,并且占用大量内存,这会影响计算效率。BM3只学习用户的ID表示,而忽略了用户的多模态表示,这是对用户多模态偏好的忽视。SLMRec通过简单聚合物品多模态特征来获取用户多模态特征,该方法不利于捕捉用户多模态偏好。同时,SLMRec基于预定义不同模态之间层次相关性生成增强视图,可能会影响其在各种多模态推荐数据集上自监督信号的有效性。VBPR采用了带有偏好向量的简单设计,更擅长处理较少的多模态特征和密集交互,当项目数量增加时,VBPR会变得不稳定。MMGCN模型采用统一的方式融合不同模态的用户偏好,忽略了用户对不同模态偏好程度的差异,每个模态均等的权重可能会导致模型性能变差。MMGCL中直接屏蔽模态特征的方法可能会导致重要信息丢失。
DPRec主要优势在于:避免了动态生成项目间的潜在结构,对于用户嵌入表示不仅学习ID表示也学习多模态表示,并通过自注意力机制学习每种模态信息对于用户和项目的重要性,考虑到了不同模态偏好程度差异。除此之外,DPRec的多模态编码层搭建了同时考虑记忆与扩展信息的双视角信息编码层。多模态对齐方面,创新性地提出了使用超图学习到的丰富嵌入表示与双视角信息编码层的嵌入表示进行多级跨模态精准对齐,将多种模态嵌入映射至同一空间之中进行融合,提高整体推荐效果。
在本节中,将通过实验分析DPRec的不同组成部分,并开发了5个变体,以更好地验证其有效性。在3个数据集上的实验结果如表4所示,DPRec优于其5个变体:DPRec-o,DPRec-e,DPRec-m,DPRec-c,DPRec-c1。这表明,通过使用LLM进行特征表示操作确实可以得到更加准确的特征表示,从而提高推荐性能。同时,仅去掉第3级别的模态对齐相比于完全去掉跨模态对齐操作性能有明显提升。这表明,利用超图学习到的嵌入信息进行多级跨模态对齐操作能有效地将不同模态映射至同一空间中。此外,还观察到,当舍弃记忆视角信息编码时,整体推荐效果出现了大幅度下滑,而舍弃扩展视角信息编码时,效果则出现了小范围的下降。这表明,利用超图学习到的嵌入信息进行多级跨模态对齐操作能有效地将不同模态映射至同一空间中。此外,还观察到,当舍弃记忆视角信息编码时,整体推荐效果出现了大幅度下滑,而舍弃扩展视角信息编码时,效果则出现了小范围的下降。这说明用户的历史交互信息仍是捕获用户兴趣偏好的重点,但通过对原始图进行一定范围的图扩展操作,确实也可以捕捉到一些用户额外感兴趣的信息和可能感兴趣的物品。
在本节中,将对DPRec方法的超参数进行分析,这些超参数包括项目邻居数量$ K $、平衡超参数$ {\lambda }_{\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{g}\mathrm{n}} $、超边数量$ N $和可调节因子$ \alpha $。
为了避免传播不相关项目信息,仅利用最相似的$ K $个项目构建项目-项目图。结果如图6所示,可以发现,对于Baby数据集和Sports数据集,$ K $=5是项目邻居数量的最佳值。然而,对于Clothing数据集,$ K $=10则是最佳值。这表明$ K $的最优值根据场景不同而异,但可以得到一个确切的结论:盲目使用较大$ K $值构建项目-项目图,会引入很多不相关的项目邻居,造成大量噪声传播,从而导致推荐结果下降。
在图7中展示的DPRec模型性能分析中,超参数$ {\lambda }_{\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{g}\mathrm{n}}\in \{0.1,0.2,0.3\} $三个值中进行选择。结果表明,除了Baby数据集的NDCG@20在$ {\lambda }_{\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{g}\mathrm{n}}=0.1 $时达到最佳外,其余所有数据集的性能评价指标均在$ {\lambda }_{\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{g}\mathrm{n}}= 0.2 $时达到最优。这表明,当$ {\lambda }_{\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{g}\mathrm{n}}=0.2 $时,模型整体性能最佳,说明模态对齐达到了最好的效果。而当$ {\lambda }_{\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{g}\mathrm{n}} $值过大时,可能会导致跨模态对齐效果欠佳,无法将多种模态嵌入有效地映射至同一空间中。
从图8(a)中可以观察到,随着超边数量从1增至4,模型性能持续提升。然而,当超边数量继续增加时,模型性能出现了轻微下降。当超边进一步提高至128条或更多时,模型性能达到最佳。这一现象表明,当超边数量较少时,模型学习到的嵌入可能不够稳定;而当超边达到一定阈值时,超图卷积学习能够捕捉到各个模态的丰富嵌入信息,通过与编码层的对齐,使其映射至同一空间,从而变得更加丰富和准确。
从图8(b)可以看出,当可调节因子$\alpha=0.4 $时,可以得到最佳的超图混合模态嵌入。随着$ \alpha $的增加,模型的整体性能开始下降。这表明适当增加$ \alpha $的值有助于获得更准确的融合嵌入,但过大的$ \alpha $可能会导致模态嵌入融合过程混乱,影响模态对齐的准确性,进而导致整体性能的降低。
本文中模型计算开销主要由3部分组成:利用LLM将文本与图像转为模态嵌入、构建超图进行消息传递,以及利用超图得到的嵌入与双视角嵌入进行的跨模态对齐操作。以Clothing数据集为例,其包括23033个项目,下面讨论其开销。
1)对于数据处理部分。利用通义千问(Qwen2.5-Max)对单条数据进行处理时,大概耗时50 ms,将所有数据处理进行批量并行处理后,总耗时为19 h左右(离线完成)。尽管LLM的数据处理成本较高,但其带来的特征表示优势,可显著提升推荐表现。
2)对于超图构建及消息传递部分。由式(15)(16)可知,超图初始化阶段的总耗时为4.3 s,其中超边依赖矩阵计算通过稀疏矩阵乘法优化后,内存占用约为7.2 GB。当使用Gumbel-Softmax进行重新参数化后,可大幅降低项目超边关联矩阵的计算复杂度。由式(17)(18)可知,消息传递阶段在dropout对矩阵密度的有效控制下耗时约为4.8 s。
3)跨模态对齐部分。其主要开销在于3级KL散度计算,其中第3级别的复杂度为前2级别的3倍,3级对齐的总耗时占单个训练epoch的5%,约为1.8 s且内存占用稳定在3.5 GB以内。
由此可见,当不将离线数据处理的时间计算在内时,整体的开销比较可观。即这种通过超图结构的多级消息传递与跨模态分布对齐的协同设计,能够在不牺牲模型表达能力的前提下有效解决多模态推荐系统中复杂关系建模的计算效率瓶颈。
本文提出了一种融合记忆与扩展的图卷积神经网络推荐模型DPRec。具体来说,首先利用LLM进行特征表示操作,然后将所得的多模态嵌入投入到经过特别设计的双视角信息编码层中。记忆视角编码通过LLM提取的实体词嵌入来丰富用户行为信息,从而获得更细粒度的记忆能力。同时,为有效解决模态空间分布不一致问题,提出了利用超图学习到的模态嵌入与双视角编码层学习到的模态嵌入进行多级跨模态精准对齐,将不同模态信息映射至同一空间中。这有助于在模态融合层获得丰富且准确的用户与项目嵌入,从而提升推荐性能。
作者贡献声明:李佚名、于之晟提出了总体思路和实验方案;于亚新提出指导意见并修改论文;司一廷完成实验并撰写论文;叶育松完成数据采集和预处理。

参考文献 引证文献
排序方式:
1
周俊, 董晓蕾, 曹珍富, 等. 推荐系统的隐私保护研究进展[J]. 计算机研究与发展, 2019, 56(10): 2033−2048
Zhou Jun, Dong Xiaolei, Cao Zhenfu. Research advances on privacy preserving in recommender systems[J]. Journal of Computer Research and Development, 2019, 56(10): 2033−2048 (in Chinese)
2
He Ruining, McAuley J. VBPR: Visual Bayesian personalized ranking from implicit feedback[C] //Proc of the 30th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2016: 144−150
3
Zhang Fuzheng, Lian Defu, Xie Xing, et al. Collaborative knowledge base embedding for recommender systems[C] //Proc of the 22nd ACM SIGKDD Int Conf on Knowledge Discovery and Data Mining. New York: ACM, 2016: 353−362
4
Koren Y, Bell R, Volinsky. Matrix factorization techniques for recommender systems[J]. IEEE Computer, 2009, 42(8): 30−37
5
Chen Lei, Wu Le, Hong Richang, et al. Revisiting graph based collaborative filtering: A linear residual graph convolutional network approach[C] //Proc of the 34th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2020: 27−34
6
Mao Kelong, Zhu Jieming, Xiao Xi, et al. UltraGCN: Ultra simplification of graph convolutional networks for recommendation[C] //Proc of the 30th ACM Int Conf on Information & Knowledge Management. New York: ACM, 2021: 1253−1262
7
Wu Shiwen, Sun Fei, Zhang Wentao, et al. Graph neural networks in recommender systems: A surery[J]. ACM Computing Surveys, 2022, 55(5): 1−37
8
Wei Yinwei, Wang Xiang, He Xiangnan, et al. MMGCN: Multi-modal graph convolution network for personalized recommendation of micro-video[C] // Proc of the 27th ACM Int Conf on Multimedia. New York: ACM, 2019: 1437−1445
9
Wei Yinwei, Wang Xiang, He Xiangnan, et al. Graph-refined convolutional network for multimedia recommendation with implicit feedback[C] //Proc of the 28th ACM Int Conf on Multimedia. New York: ACM, 2020: 3541−3549
10
Wang Qifan, Wei Yinwei, Yin Jianhua, et al. DualGNN: Dual graph neural network for multimedia recommendation[J]. IEEE Transactions on Multimedia, 2021, 25(25): 1074−1084
11
Pagliardini M, Gupta P, Jaggi M. Unsupervised learning of sentence embeddings using compositional n-gram features [C] //Proc of the 2018 Conf of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg, PA: ACL, 2018: 528−540
12
Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition[C/OL] //Proc of the 3rd Int Conf on Learning Representations. 2014[2024-05-17]. https://www.semanticscholar.org/reader/eb42cf88027de515750f230b23b1a057dc782108
13
Lin Guojiao, Zhen Meng, Wang Dongjie, et al. GUME: Graphs and user modalities enhancement for long-tail multimodal recommendation[C] //Proc of the 33rd ACM Int Conf on Information and Knowledge Management. New York: ACM, 2024: 1400−1409
14
舒文韬, 李睿潇, 孙天祥, 等. 大型语言模型: 原理、实现与发展[J]. 计算机研究与发展, 2024, 61(2): 351−361
Shu Wentao, Li Ruixiao, Sun Tianxiang, et al. Large language models: Principles, implementation, and progress[J]. Journal of Computer Research and Development, 2024, 61(2): 351−361(in Chinese)
15
Devlin J, Chang Mingwei, Lee K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding[C] //Proc of the 2019 Conf of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg, PA: ACL, 2019: 4171−4186
16
Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C] //Proc of the 31st Int Conf on Neural Information Processing Systems. Long Beach, CA: NeurIPS, 2017: 6000−6010
17
Wei wei, Ren Xubin, Tang Jiabin, et al. LLMRec: Large language models with graph augmentation for recommendation[C] //Proc pf the 17th ACM Int Conf on Web Search and Data Mining. New York: ACM, 2024: 806−815
18
Tan Juntao, Xu Shuyuan, Hua Yingqiang, et al. IDGenRec: LLM-RecSys alignment with textual id learning[C] //Proc of the 47th Int ACM Conf on Research and Development in Information Retrieval. New York: ACM, 2024: 355−364
19
Bao Keqin, Zhang Jizhi, Zhang Yang, et al. TALLRec: An effectiveand efficient tuning framework to align large language model with recommendation[C] //Proc of the 17th ACM Conf on Recommender Systems. New York: ACM, 2023: 1007−1014
20
Wang Suhang, Wang Yilin, Tang Jiliang, et al. What your images reveal: Exploiting visual contents for point-of-interest recommendation[C] //Proc of the 26th Int Conf on World Wide Web. New York: ACM, 2017: 391−400
21
Chen Jingyuan, Zhang Hanwang, He Xiangnan, et al. Attentive collaborative filtering: Multimedia recommendation with item-and component-level attention[C] //Proc of the 40th Int ACM SIGIR Conf on Research and Development in Information Retrieval. New York: ACM, 2017: 335−344
22
Zhang Jinghao, Zhu Yanqiao, Liu Qiang, et al. Mining latent structures for multimedia recommendation[C] //Proc of the 29th ACM Int Conf on Multimedia. New York: ACM, 2021: 3872−3880
23
Zhou Xin, Shen Zhiqi. A tale of two graphs: Freezing and denoising graph structures for multimodal recommendation[C] //Proc of the 31st ACM Int Conf on Multimedia. New York: ACM, 2023: 935−943
24
Sun Rui, Cao Xuezhi, Zhao Yan, et al. Multi-modal knowledge graphs for recommender systems[C] //Proc of the 29th ACM Int Conf on Information & Knowledge Management. New York: ACM, 2020: 1405−1414
25
林晶晶, 冶忠林, 赵海兴, 等. 超图神经网络综述[J]. 计算机研究与发展, 2024, 61(2): 362−384
Lin Jingjing, Ye Zhonglin, Zhao Haixing, et al. Survey on hypergraph nerual networks[J]. Journal of Computer Research and Development, 2024, 61(2): 362−384(in Chinese)
26
Feng Yifan, You Haoxuan, Zhang Zizhao, et al. Hypergraph neural networks[C] //Proc of the 33rd AAAI Conf on Artificial Intelligence and 31st Innovative Applications of Artificial Intelligence Conf and 9th AAAI Symp on Educational Advances in Artificial Intelligence. Palo Alto, CA: AAAI, 2019: 3558−3565
27
Jang E, Gu Shixiang, Poole B. Categorical reparameterization with Gumbel-Softmax[C/OL] //Proc of the 5th Int Conf Learning Representations. 2016[2023-03-19]. https://www.researchgate.net/publication/309663606_Categorical_Reparameterization_with_Gumbel-Softmax
28
Xia Lianghao, Huang Chao, Zhang Chuxu. Self-supervised hypergraph transformer for recommender systems[C] //Proc of the 28th ACM SIGKDD Conf on Knowledge Discovery and Data Mining. New York: ACM, 2022: 2100−2109
29
Guo Zhiqiang, Li Jianjun, Li Guohui, et al. LGMRec: Local and global graph learning for multimodal recommendation[C] //Proc of the 38th AAAI Conf on Artificial Intelligence and 36th Conf on Innovative Applications of Artificial Intelligence and 14th Symp on Educational Advances in Artificial Intelligence. Palo Alto, CA: AAAI, 2024: 8454−8462
30
Xu Jinfeng, Chen Zheyu, Yang Shuo, et al. MENTOR: Multi-level self-supervised learning for multimodal recommendation[C] //Proc of the 39th Annual AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2025: 12908−12917
31
McAuley J, Targett C, Shi Qinfeng, et al. Image-based recommendations on styles and substitutes[C] //Proc of the 38th Int ACM SIGIR Conf on Research and Development in Information Retrieval. New York: ACM, 2015: 43−52
32
Rendle S, Freudenthaler C, Gantner Z, et al. BPR: Bayesian personalized ranking from implicit feedback[C] //Proc of the 25th Conf on Uncertainty in Artificial Intelligence. Palo Alto, CA: AUAI, 2009: 452−461
33
He Xiangnan, Deng Kuan, Wang Xiang, et al. LightGCN: Simplifying and powering graph convolution network for recommendation[C] //Proc of the 43rd Int ACM SIGIR Conf on Research and Development in Information Retrieval. New York: ACM, 2020: 639−648
34
Lin Zihan, Tian Changxin, Hou Yupeng, et al. Improving graph collaborative filtering with neighborhood-enriched contrastive learning[C] //Proc of the ACM Web Conf. New York: ACM, 2022: 2320−2329
35
Yi Zixuan, Wang Xi, Ounis I, et al. Multi-modal graph contrastive learning for micro-video recommendation[C] //Proc of the 45th Int ACM SIGIR Conf on Research and Development in Information Retrieval. New York: ACM, 2022: 1807−1811
36
Tao Zhulin, Liu Xiaohao, Xia Yewei, et al. Self-supervised learning formultimedia recommendation[J]. IEEE Transactions on Multimedia, 2022, 25(25): 5107−5116
37
Zhou Xin, Zhou Hongyu, Liu Yong, et al. Bootstrap latent representations for multi-modal recommendation[C] //Proc of the ACM Web Conf. New York: ACM, 2023: 845−854
38
Jiang Yangqin, Xia Lianghao, Wei Wei, et al. DiffMM: Multi-modal diffusion model for recommendation[C] //Proc of the 32nd ACM Conf on Multimedia. New York: ACM, 2024: 7591−7599
39
Glorot X, Bengio Y. Understanding the difficulty of training deep feedforward neural networks[C] //Proc of the 13th Int Conf on Artificial Intelligence and Statistics. Brookline, MA: JMLR. org, 2010: 249−256
40
Kingma D, Ba J. Adam: A method for stochastic optimization[C/OL] //Proc of the 3rd Int Conf on Learning Representations. 2014[2024-04-12]. https://www.semanticscholar.org/reader/a6cb366736791bcccc5c8639de5a8f9636bf87e8
2026年第63卷第1期
321
158
0
引用本文
BibTeX
文章信息
doi: 10.7544/issn1000-1239.202550040
  • 接收时间:2025-01-15
  • 首发时间:2026-11-06
  • 出版时间:2026-01-01
补充材料
相关文章
文章信息
作者
出版历史
  • 收稿日期:2025-01-15
  • 修回日期:2025-08-27
  • 录用日期:2025-09-02
基金
作者信息
    东北大学计算机科学与工程学院 沈阳 110169
    医学影像智能计算教育部重点实验室(东北大学) 沈阳 110169

通讯作者:

于亚新()
参考文献
分享链接
https://castjournals.cast.org.cn/joweb/jsjyjyfz/CN/10.7544/issn1000-1239.202550040
分享至
全文二维码

扫描看全文

引用本文
BibTeX
本文的引用情况
2种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏