收藏切换
Recent advances in open-vocabulary semantic segmentation
收藏切换
Mingyi Liu1, Guosen Xie1, *, Xiangbo Shu1, Lei Zhang2
Journal of Image and Graphics | 2026, 31(9) : 3156 - 3168
Less
收藏切换
Journal of Image and Graphics | 2026, 31(9): 3156-3168
• The intelligent empowerment of large models in the aviation field •
Recent advances in open-vocabulary semantic segmentation
Full
Mingyi Liu1, Guosen Xie1, *, Xiangbo Shu1, Lei Zhang2
Affiliations
  • 1 School of Computer Science and Engineering,Nanjing University of Science and Technology,Nanjing 210094,China
  • 2 School of Computer Science,Northwestern Polytechnical University,Xi’an 710129,China
Published: 2026-09-16 doi: 10.11834/jig.260205
Outline
收藏切换

Semantic segmentation is a fundamental task in computer vision that aims to assign a semantic label to each pixel in an image. Most conventional semantic segmentation methods are designed for a closed-set setting in which the categories involved in testing are predefined and largely consistent with those observed during training. Although this setting facilitates the learning of stable category representations and clear decision boundaries,it limits the applicability of segmentation models in real-world scenarios,where unseen objects and novel semantic concepts are frequently encountered. In this context,open-vocabulary semantic segmentation (OVSS) has emerged as an important research direction for extending semantic segmentation from closed-set prediction to open-world visual understanding. OVSS enables models to identify and segment pixel-level regions corresponding to arbitrary textual concepts by using natural language descriptions as category definitions,thereby alleviating the dependence on fixed label spaces. The rapid development of vision-language pre-trained models,especially contrastive language-image pre-training (CLIP) and related large-scale cross-modal models,has provided an important foundation for OVSS. Trained on massive image-text pairs,these models learn aligned visual and textual representations and exhibit strong transferability in open-category recognition. However,most existing vision-language models are mainly optimized for image-level semantic alignment and global visual recognition. Thus,these models have difficulty meeting the requirements of pixel-level dense prediction directly,such as accurate localization,boundary delineation,and fine-grained category discrimination. Therefore,how to effectively transfer image-level open-vocabulary recognition ability to pixel-level segmentation remains a key challenge in OVSS. This paper summarizes recent progress in OVSS from the perspectives of task background,representative methods,benchmark datasets,evaluation metrics,remote sensing extension,and future directions. First,the paper introduces the task background and basic characteristics of OVSS,after which its relationship with conventional semantic segmentation and zero-shot semantic segmentation is clarified. Compared with conventional semantic segmentation,OVSS emphasizes generalization beyond predefined categories,whereas when compared with zero-shot semantic segmentation,it further benefits from large-scale image-text pre-training and allows more flexible category definitions through natural language descriptions. Second,representative OVSS methods are reviewed in accordance with several major research routes,including zero-shot semantic segmentation,early explorations based on image-text supervision,two-stage methods,and single-stage methods. Early zero-shot segmentation methods mainly rely on semantic embeddings to transfer knowledge from seen categories to unseen categories,but their expressive ability is often limited by static semantic representations. Meanwhile,initial studies based on image-text supervision explore whether transferable semantic region representations can be learned without dense pixel annotations,thus providing important inspiration for subsequent OVSS methods. Two-stage methods usually decompose the task into class-agnostic region generation and open-vocabulary region recognition. While this paradigm is clear and modular,its performance is highly dependent on the quality of candidate regions and may incur additional computational cost. In contrast,single-stage methods aim to integrate region modeling,vision-language alignment,and pixel-level prediction within a unified framework,thereby reducing cross-stage error accumulation and improving inference efficiency. Nevertheless,they still face challenges in fine-grained localization,dense semantic alignment,and robust generalization to unseen categories. Apart from analyzing the main ideas,technical characteristics,advantages,and limitations of these methods,this paper discusses the extension of OVSS to remote sensing imagery. Compared with natural images,remote sensing images typically exhibit top-down perspectives,large variations in object scale,significant orientation changes,complex backgrounds,and cross-region distribution shifts,all of which make it difficult to directly transfer OVSS methods designed for natural-image scenarios to remote sensing applications. Therefore,recent studies have begun to explore open-vocabulary remote sensing segmentation frameworks,dedicated benchmark datasets,training-free strategies,and multimodal fusion methods. This development indicates that OVSS is gradually expanding from general natural-image understanding to more complex specialized visual domains. Commonly used datasets and evaluation metrics in OVSS are also summarized in this paper. Existing studies typically use COCO-Stuff as the main training benchmark and evaluate model generalization on datasets,such as Pascal VOC,Pascal Context,and ADE20K,under different vocabulary settings. At present,OVSS evaluation relies on conventional semantic segmentation metrics,especially mean intersection over union (mIoU). However,such metrics emphasize strict category matching and are insufficient for measuring semantic similarity,category hierarchy,and open-category generalization. Therefore,constructing evaluation protocols that better reflect the semantic openness of OVSS remains an important issue. Finally,the paper summarizes the major challenges in current OVSS research and discusses promising future directions,including the effective transfer of open-vocabulary recognition capabilities to precise pixel-level prediction,adaptive region generation,evaluation protocols tailored to open-vocabulary settings,learning with limited supervision,training-free inference,and extensions to domain-specific applications.

open-vocabulary semantic segmentation(OVSS)  /  vision-language pre-training  /  contrastive language-image pre-training(CLIP)  /  cross-category generalization  /  open-world visual understanding
刘明怡, 谢国森, 舒祥波, 张磊. 开放词汇语义分割研究进展. 中国图象图形学报, 2026 , 31 (9) : 3156 -3168 . DOI: 10.11834/jig.260205
Mingyi Liu, Guosen Xie, Xiangbo Shu, Lei Zhang. Recent advances in open-vocabulary semantic segmentation[J]. Journal of Image and Graphics, 2026 , 31 (9) : 3156 -3168 . DOI: 10.11834/jig.260205
  • National Natural Science Foundation of China(62276134)
Year 2026 volume 31 Issue 9
24
12
Cite this Article
BibTeX
Article Info
doi: 10.11834/jig.260205
  • Receive Date:2026-04-16
  • Online Date:2026-10-30
  • Published:2026-09-16
Article Data
Affiliations
History
  • Received:2026-04-16
  • Revised:2026-05-28
Funding
National Natural Science Foundation of China(62276134)
Affiliations
    1 School of Computer Science and Engineering,Nanjing University of Science and Technology,Nanjing 210094,China
    2 School of Computer Science,Northwestern Polytechnical University,Xi’an 710129,China
References
Share
https://castjournals.cast.org.cn/joweb/zgtxtxxb/EN/10.11834/jig.260205
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT