Objective To construct HerbVLM, a vision-text fusion model for the identification of traditional Chinese medicinal materials, aiming to improve intelligent recognition performance in distinguishing visually similar categories and generalizing across scenarios. Methods Based on a vision-language pretrained dual-encoder framework, a traditional Chinese medicine knowledge-enhanced prompt module (TKPM) was introduced on the text side, while a hierarchical image adaptation and de-confusion module (HIA) was constructed on the image side. Dynamic fusion and joint loss optimization were further incorporated. Comparative and ablation experiments were conducted on the open-source Chinese-Medicine-163 dataset, and cross-domain and few-shot adaptation validation was performed on a self-collected external dataset of the “new Zhebawei”. Results On the test set, HerbVLM achieved top-1 accuracy (Top-1 Acc), macro precision (P_macro), macro recall (R_macro), and macro F1-score (F1_macro) of 98.69%, 98.70%, 98.71%, and 98.70%, respectively, outperforming comparative methods including residual network (ResNet), context optimization (CoOp), conditional context optimization (CoCoOp), training-free CLIP-Adapter (Tip-Adapter), multi-modal prompt learning (MaPLe), and contrastive vision-language pretraining in Chinese (Chinese-CLIP). Ablation experiments showed that text enhancement, image adaptation, de-confusion constraints, and dynamic fusion all brought stable performance gains. In external testing on the self-collected Zhejiang Eight Flavors dataset, the recognition rates for known categories Lingzhi (Ganoderma), Fupenzi (Rubi Fructus), Tiepishihu (Dendrobii Officinalis Caulis), and Qianhu (Peucedani Radix) were 96.00%, 100.00%, 100.00%, and 100.00%, respectively. For unknown categories, the average accuracies under 1-shot, 2-shot, and 5-shot settings were approximately 84.00%, 91.54%, and 98.20%, respectively, demonstrating favorable few-shot transfer capability. Conclusion HerbVLM significantly improves the fine-grained recognition accuracy and cross-domain generalization performance of traditional Chinese medicinal materials through multimodal collaborative modeling of image-text knowledge, and can provide methodological support for traditional Chinese medicinal material teaching, quality inspection assistance.

traditional Chinese medicinal materials identification  /  vision-text fusion  /  multimodality  /  prompt learning  /  few-shot learning  /  cross-domain generalization