收藏切换
A Multimodal Adaptive Entity Recognition Method Based on Reinforcement Strategy Feedback
收藏切换
MingHai Jiao, Benhang Fan, Jing Wang, Yuhuai Peng*
Journal of Computer Research and Development | 2026, 63(2) : 294 - 304
Less
收藏切换
Journal of Computer Research and Development | 2026, 63(2): 294-304
• Artificial Intelligence •
A Multimodal Adaptive Entity Recognition Method Based on Reinforcement Strategy Feedback
Full
MingHai Jiao, Benhang Fan, Jing Wang, Yuhuai Peng*
Affiliations
  • School of Computer Science and Engineering, Northeastern University, Shenyang 110167
  • born in 1973. PhD, associate professor, master supervisor. Member of CCF. His main research interests include artificial intelligence, image recognition, and big data parsing and optimization

    born in 2002. Master candidate. His main research interests include machine learning and multimodal information extraction

    born in 1992. PhD candidate. Her main research interests include Internet of things and digital twin

    born in 1983. PhD, professor, PhD supervisor. His main research interests include Internet of things, cyber physical systems, and intelligent information networks

About Author:

born in 1973. PhD, associate professor, master supervisor. Member of CCF. His main research interests include artificial intelligence, image recognition, and big data parsing and optimization

born in 2002. Master candidate. His main research interests include machine learning and multimodal information extraction

born in 1992. PhD candidate. Her main research interests include Internet of things and digital twin

born in 1983. PhD, professor, PhD supervisor. His main research interests include Internet of things, cyber physical systems, and intelligent information networks

Published: 2026-02-01 doi: 10.7544/issn1000-1239.202550273
Outline
收藏切换

The core objective of the named entity recognition (NER) is to identify entities with specific semantic categories and types in unstructured text. With the rapid growth of social media, textual information is often accompanied by visual content, forming multimodal data. To improve the accuracy of entity recognition, multimodal NER (MNER) methods exploit semantic information from different modalities fully to achieve the complementary and deep fusion of cross-modal features. However, differences in the representation of modalities may introduce visual noise that interferes with entity recognition. Issues such as entity ambiguity or contextual semantic vagueness within the textual modality complicate recognition. To address these challenges, we propose a multimodal NER method based on a reinforcement strategy feedback with adaptive loss mechanism. First, the method adopts a three-stage chain of thought (COT) reasoning process based on GPT-4o to form a progressive reasoning framework that incorporates the adaptive feedback mechanism in reinforcement learning. The degree of matching between images and text is scored, and the interference of visual noise is effectively filtered using an adaptive decision function. Second, four task-specific loss functions are designed and jointly optimized through an adaptive weighted fusion strategy to alleviate the uncertainty caused by contextual ambiguity. Experiments on two representative public datasets (Twitter-2015 and Twitter-2017) show that the overall F1 scores of our proposed method are 86.45% and 93.80%, respectively, representing a significant improvement on current state-of-the-art baseline models.

named entity recognition  /  unstructured text  /  multimodal data  /  reinforcement learning  /  adaptive loss  /  chain of thought
焦明海, 樊本航, 王静, 彭玉怀. 基于强化策略反馈的多模态自适应实体识别方法. 计算机研究与发展, 2026 , 63 (2) : 294 -304 . DOI: 10.7544/issn1000-1239.202550273
MingHai Jiao, Benhang Fan, Jing Wang, Yuhuai Peng. A Multimodal Adaptive Entity Recognition Method Based on Reinforcement Strategy Feedback[J]. Journal of Computer Research and Development, 2026 , 63 (2) : 294 -304 . DOI: 10.7544/issn1000-1239.202550273
Year 2026 volume 63 Issue 2
200
104
Cite this Article
BibTeX
Article Info
doi: 10.7544/issn1000-1239.202550273
  • Receive Date:2025-04-21
  • Online Date:2026-11-06
  • Published:2026-02-01
Article Data
Affiliations
History
  • Received:2025-04-21
  • Revised:2025-08-07
  • Accepted:2025-09-02
Affiliations
    School of Computer Science and Engineering, Northeastern University, Shenyang 110167
References
Share
https://castjournals.cast.org.cn/joweb/jsjyjyfz/EN/10.7544/issn1000-1239.202550273
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT