中草药

HerbVLM：一种基于图文提示增强与视觉去混淆的中药材鉴别模型

PDF下载

中草药 | 数据挖掘与循证医学 2026,57(19): 7722-7738

收起

中草药 |数据挖掘与循证医学 2026 , 57 (19) : 7722 -7738

HerbVLM：一种基于图文提示增强与视觉去混淆的中药材鉴别模型

全屏

叶诚¹, 陈天雪², 于哲轩², 赵希曦³, 虞立², 金伟¹

作者信息

通讯作者:

虞立

作者简介:

叶诚: 叶诚（2005—），本科生，浙江省大学生新苗人才计划项目负责人，主要从事图像算法研究。E-mail:yecheng_yygz@163.com

HerbVLM: A traditional Chinese medicinal materials identification model based on vision-text prompt enhancement and visual deconfusion

YE Cheng¹, CHEN Tianxue², YU Zhexuan², ZHAO Xixi³, YU Li², JIN Wei¹

Affiliations

doi: 10.7501j.issn.0253-2670.2026.19.018

文章导航

摘要

收起

目的构建一种面向中药材鉴别的视觉-文本融合模型HerbVLM，提升中药材智能识别在相似类别区分与跨场景泛化中的性能。方法基于视觉-语言预训练双编码器框架，在文本侧引入中药知识增强提示模块（traditional Chinese medicine knowledge-enhanced prompt module，TKPM），在图像侧构建分层适配与去混淆模块（hierarchical image adaptation and de-confusion module，HIA），并结合动态融合与联合损失进行优化。在Chinese-Medicine-163开源数据集上开展对比与消融实验，并在自采“新浙八味”外部数据集上进行跨域与小样本适配验证。结果在测试集上，HerbVLM的首位预测准确率（top-1 accuracy，Top-1 Acc）、宏平均精确率（macro precision，Pmacro）、宏平均召回率（macro recall，Rmacro）、宏平均F1值（macro F1-score，F1macro）分别达到98.69%、98.70%、98.71% 和98.70%，优于残差网络（residual network，ResNet）、上下文优化（context optimization，CoOp）、条件上下文优化（conditional context optimization，CoCoOp）、免训练CLIP适配器（training-free CLIP-Adapter，Tip-Adapter）、多模态提示学习（multi-modal prompt learning，MaPLe）、中文视觉-语言对比预训练模型（contrastive vision-language pretraining in Chinese，Chinese-CLIP）等对比方法。消融实验显示，文本增强、图像适配、去混淆约束和动态融合均带来稳定增益。在自采浙八味数据集外部测试中，已知类别灵芝、覆盆子、铁皮石斛、前胡识别率分别为96.00%、100.00%、100.00%和100.00%；未知类别在1-shot、2-shot、5-shot条件下平均准确率分别约为84.00%、91.54%和98.20%，显示出较好的小样本迁移能力。结论 HerbVLM通过图文知识多模态协同建模显著提升了中药材细粒度识别精度与跨域泛化性能，可为中药材教学、质检辅助等场景提供方法支持。

关键词

中药材鉴别 / 视觉-文本融合 / 多模态 / 提示学习 / 小样本学习 / 跨域泛化

Abstract

收起

Objective To construct HerbVLM, a vision-text fusion model for the identification of traditional Chinese medicinal materials, aiming to improve intelligent recognition performance in distinguishing visually similar categories and generalizing across scenarios. Methods Based on a vision-language pretrained dual-encoder framework, a traditional Chinese medicine knowledge-enhanced prompt module (TKPM) was introduced on the text side, while a hierarchical image adaptation and de-confusion module (HIA) was constructed on the image side. Dynamic fusion and joint loss optimization were further incorporated. Comparative and ablation experiments were conducted on the open-source Chinese-Medicine-163 dataset, and cross-domain and few-shot adaptation validation was performed on a self-collected external dataset of the “new Zhebawei”. Results On the test set, HerbVLM achieved top-1 accuracy (Top-1 Acc), macro precision (P_macro), macro recall (R_macro), and macro F1-score (F1_macro) of 98.69%, 98.70%, 98.71%, and 98.70%, respectively, outperforming comparative methods including residual network (ResNet), context optimization (CoOp), conditional context optimization (CoCoOp), training-free CLIP-Adapter (Tip-Adapter), multi-modal prompt learning (MaPLe), and contrastive vision-language pretraining in Chinese (Chinese-CLIP). Ablation experiments showed that text enhancement, image adaptation, de-confusion constraints, and dynamic fusion all brought stable performance gains. In external testing on the self-collected Zhejiang Eight Flavors dataset, the recognition rates for known categories Lingzhi (Ganoderma), Fupenzi (Rubi Fructus), Tiepishihu (Dendrobii Officinalis Caulis), and Qianhu (Peucedani Radix) were 96.00%, 100.00%, 100.00%, and 100.00%, respectively. For unknown categories, the average accuracies under 1-shot, 2-shot, and 5-shot settings were approximately 84.00%, 91.54%, and 98.20%, respectively, demonstrating favorable few-shot transfer capability. Conclusion HerbVLM significantly improves the fine-grained recognition accuracy and cross-domain generalization performance of traditional Chinese medicinal materials through multimodal collaborative modeling of image-text knowledge, and can provide methodological support for traditional Chinese medicinal material teaching, quality inspection assistance.

Key words

traditional Chinese medicinal materials identification / vision-text fusion / multimodality / prompt learning / few-shot learning / cross-domain generalization

引用本文

叶诚, 陈天雪, 于哲轩, 赵希曦, 虞立, 金伟. HerbVLM：一种基于图文提示增强与视觉去混淆的中药材鉴别模型. 中草药, 2026 , 57 (19) : 7722 -7738 . DOI: 10.7501j.issn.0253-2670.2026.19.018

YE Cheng, CHEN Tianxue, YU Zhexuan, ZHAO Xixi, YU Li, JIN Wei. HerbVLM: A traditional Chinese medicinal materials identification model based on vision-text prompt enhancement and visual deconfusion[J]. Chinese Traditional and Herbal Drugs, 2026 , 57 (19) : 7722 -7738 . DOI: 10.7501j.issn.0253-2670.2026.19.018

基金

收起

浙江省自然科学基金项目/优青项目 (LZYQ25H270001); 浙江省自然科学基金项目/探索一般 (LY24H270007); 浙江省中医药管理局中医药健康服务研究计划项目 (2026ZF41); 2026年浙江省大学生科技创新活动计划新苗人才计划项目 (2026R410A003)

参考文献

收起

文献

收起

参考文献引证文献

排序方式：

Tian D C, Zhou C, Wang Y, et al. NB-TCM-CHM: Image dataset of the Chinese herbal medicine fruits and its application in classification through deep learning [J]. Data Brief, 2024, 54: 110405.
张谊, 万华, 涂淑琴. 基于计算机视觉的中药饮片分类技术综述与案例研究[J]. 计算机应用, 2022, 42(10): 3224-3234.
孙兴, 李杨, 毛天驰, 等. 改进YOLOv8在中药饮片自动识别标注中的研究与应用[J]. 中草药, 2025, 56(1): 216-226.
Miao J F, Huang Y N, Wang Z S, et al. Image recognition of traditional Chinese medicine based on deep learning [J]. Front Bioeng Biotechnol, 2023, 11: 1199803.
Ji H H, Liu X J, Wang L, et al. Image recognition of Chinese herbal medicine using adaptive gamma correction based on convolutional neural network [C] //2024 IEEE 13th Data Driven Control and Learning Systems Conference (DDCLS). Kaifeng, China: IEEE, 2024: 1428-1433.
Wang H, Chen Z, Huo J, et al. Research on micro-morphological feature recognition method of traditional Chinese medicinal herbs based on deep convolutional neural network [J]. J Combinat Mathemat Combinat Comput, 2025, 127B: 7759-7776.
王双双, 吴纯洁, 艾莉, 等. 自动识别技术在中药饮片行业的展望[J]. 中药与临床, 2012, 3(3): 52-55.
Datsi T, Aznag K, BenAli B A, et al. A short survey on multimodal data fusion in image classification [C] //2024 International Conference on Global Aeronautical Engineering and Satellite Technology (GAST). Marrakesh, Morocco: IEEE, 2024: 1-4.
Zhang T B, Xu H R, Wang Z Y, et al. Deep learning for subtype recognition of Yang deficiency tongue images in traditional Chinese medicine [J]. Digital Chin Med, 2026, 9(2): 197-210.
Lu X, Feng Y, Jia X D, et al. Multi-label tongue image classification via label semantics-driven feature interaction enhancement [J]. Comput Biol Chem, 2026, 120: 108735.
Chen J Y, Cai Z Y, Liu Z H, et al. ShizhenGPT: Towards multimodal LLMs for traditional Chinese medicine [EB/OL]. arXiv [2025-08-20]. https://arxiv.org/abs/2508. 14706
Li Q, Zhao F, Zhao L L, et al. A system of multimodal image-text retrieval based on pre-trained models fusion [J]. Concurr Comput Pract Exp, 2025, 37(3): e8345.
Shen Q, Ruan C G, Yang L. Multi-granularity knowledge embedding for multi-label herbal medicine recognition [J]. Neurocomputing, 2025, 649: 130707.
Li S, Liu F, Hao Z H, et al. Logits DeConfusion with CLIP for few-shot learning [C] //2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Nashville, TN, USA: IEEE, 2025: 25411-25421.
飞桨AI Studio. 智慧岐黄：163种中药材深度图像辨识库[EB/OL]. [2026-02-15]. https://aistudio.baidu.com/datasetdetail/246739.
Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision [EB/OL]. arXiv [2021-02-26]. https://arxiv.org/abs/2103.00020.
Zhou K Y, Yang J K, Loy C C, et al. Learning to prompt for vision-language models [J]. Int J Comput Vis, 2022, 130(9): 2337-2348.
Zhou K Y, Yang J K, Loy C C, et al. Conditional prompt learning for vision-language models [C] //2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans, LA, USA: IEEE, 2022: 16795-16804.
Pratt S, Covert I, Liu R, et al. What does a Platypus Look like? Generating customized prompts for zero-shot image classification [C] //2023 IEEE/CVF International Conference on Computer Vision (ICCV). Paris, France: IEEE, 2024: 15645-15655.
Gao P, Geng S J, Zhang R R, et al. CLIP-adapter: Better vision-language models with feature adapters [J]. Int J Comput Vis, 2024, 132(2): 581-595.
Zhang R R, Zhang W, Fang R Y, et al. Tip-adapter: Training-free adaption of CLIP for few-shot classification [C] //Computer Vision—ECCV 2022. Cham: Springer, 2022: 493-510.
Khattak M U, Rasheed H, Maaz M, et al. MaPLe: Multi-modal prompt learning [C] //2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Vancouver, BC, Canada: IEEE, 2023: 19113-19122.
Cherti M, Beaumont R, Wightman R, et al. Reproducible scaling laws for contrastive language-image learning [C] //2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Vancouver, BC, Canada: IEEE, 2023: 2818-2829.
Yang A, Pan J S, Lin J Y, et al. Chinese CLIP: Contrastive vision-language pretraining in Chinese [EB/OL]. arXiv [2022-11-02]. https://arxiv.org/abs/2211.01335
He K M, Zhang X Y, Ren S Q, et al. Deep residual learning for image recognition [C] //2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV, USA: IEEE, 2016: 770-778.
Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16x16 words: Transformers for image recognition at scale [EB/OL]. arXiv [2020-10-22]. https://arxiv.org/abs/2010.11929.
Zeng J, Wu H, He M. Image classification combined with faster R-CNN for the peak detection of complex components and their metabolites in untargeted LC-HRMS data [J]. Anal Chim Acta, 2023, 1238: 340189.
韩勇, 兰杰, 郭瑞瑶, 等. 基于深度学习的中药材饮片识别[J]. 西北农业学报, 2023, 32(11): 1859-1867.
Xu Y X, Wen G H, Hu Y, et al. Multiple attentional pyramid networks for Chinese herbal recognition [J]. Pattern Recognit, 2021, 110: 107558.
Sun X, Qian H N. Chinese herbal medicine image recognition and retrieval by convolutional neural network [J]. PLoS One, 2016, 11(6): e0156327.
Sun X, Qian H N, Xiong Y L, et al. Deep learning-enabled mobile application for efficient and robust herb image recognition [J]. Sci Rep, 2022, 12: 6579.
Kawadkar K. Comparative analysis of vision transformers and convolutional neural networks for medical image classification [EB/OL]. arXiv [2025-07-24]. https://arxiv.org/abs/2507.21156.
Jung D H, Kim H Y, Won J H, et al. Development of a classification model for Cynanchum wilfordii and Cynanchum auriculatum using convolutional neural network and local interpretable model-agnostic explanation technology [J]. Front Plant Sci, 2023, 14: 1169709.
Dai W B, Ma Y X, Fan Y, et al. A multi-scale feature extraction algorithm for Chinese herbal medicine image classification [J]. Appl Sci, 2025, 15(8): 4271.
Pan D P, Guo Y L, Fan Y F, et al. Development and application of traditional Chinese medicine using AI machine learning and deep learning strategies [J]. Am J Chin Med, 2024, 52(3): 605-623.

2026年第57卷第19期

PDF下载

引用本文

BibTeX

文章信息

doi: 10.7501j.issn.0253-2670.2026.19.018

接收时间：2026-06-05
首发时间：2026-10-29

补充材料

相关文章

文章信息

作者

出版历史

收稿日期：2026-06-05

基金

作者信息

参考文献

分享链接

https://castjournals.cast.org.cn/joweb/zcy/CN/10.7501j.issn.0253-2670.2026.19.018

分享至

全文二维码

扫描看全文

引用本文

BibTeX

本文的引用情况

2种不同金属材料的力学参数

科 Family	属数 Number of genus	种数 Number of species	占总种数比例 Percentage of total species (%)	属 Genus	种数 Number of species	占总种数比例 Percentage of total species (%)
鹅膏菌科Amanitaceae	2	11	5.26	鹅膏菌属 Amanita	10	4.78
小菇科 Mycenaceae	2	12	5.74	丝盖伞属 Inocybe	5	2.39
多孔菌科 Polyporaceae	8	14	6.70	蜡蘑属 Laccaria	5	2.39
红菇科 Russulaceae	3	23	11.00	小皮伞属 Marasmius	6	2.87
				小菇属 Mycena	11	5.26
				光柄菇属 Pluteus	5	2.39
				红菇属 Russula	17	8.13
				栓菌属 Trametes	5	2.39

关闭全屏

BibTeX
EndNote
RefWorks
TxT