收藏切换
Semantic alignment and locality-driven open-vocabulary semantic segmentation
收藏切换
Xinjing Wang, Ying Gao*, Yaqi Zou, Zhengyu Zhu, Chunxue Xu, Qi Zhao
Journal of Image and Graphics | 2026, 31(8) : 2774 - 2784
Less
收藏切换
Journal of Image and Graphics | 2026, 31(8): 2774-2784
• Embodied Intelligence and Humanoid Robotics Frontiers •
Semantic alignment and locality-driven open-vocabulary semantic segmentation
Full
Xinjing Wang, Ying Gao*, Yaqi Zou, Zhengyu Zhu, Chunxue Xu, Qi Zhao
Affiliations
  • School of Data Science,Qingdao University of Science and Technology,Qingdao 266000,China
Published: 2026-08-16 doi: 10.11834/jig.250540
Outline
收藏切换
Objective

Accurate and fine-grained scene understanding remains a key challenge for embodied intelligence and humanoid robotic systems,which require real-time and robust perception. In particular,when applying vision-language models to pixel-level segmentation,issues,such as localization blur and sensitivity to scale and viewpoint,often impede reliable spatial reasoning. Open-vocabulary semantic segmentation (OVSS) offers a promising solution by enabling pixel-level recognition of arbitrary textual categories without relying on exhaustive pixel-wise annotations. However,large-scale vision-language models (VLMs),such as CLIP,are primarily optimized for image- or region-level tasks,resulting in limited spatial granularity and degraded dense prediction performance. This study proposes a training-free inference paradigm that enhances the pixel-wise discriminability and engineering readiness of VLM-based OVSS frameworks without modifying pretrained model weights,thereby improving their applicability to real-time robotic perception and decision-making and addressing these limitations.

Method

We introduce TG-CLIP,a practical framework built on a frozen CLIP (ViT-B/16) backbone,incorporating two lightweight and complementary inference-stage operators. These operators inject text-conditioned semantic bias into local visual representations and amplify prediction robustness across scales and viewpoints. The first operator,text-guided recalibration (TGR),treats text queries as conditional semantic probes: given a set of class queries encoded into normalized query vectors,TGR computes per-location query-visual affinities and back-projects the resulting semantic signal into the local feature space. The injected signal is blended with the original visual features under a tunable injection strength and renormalized to ensure that the final logits retain a cosine-similarity interpretation. Moreover,TGR is a parameter-free forward operator that requires no pixel-level supervision or fine-tuning,solely relying on the pretrained visual and textual encoders to perform a semantic recalibration of local responses. The second operator,multi-view consistency reasoning (MCR),aggregates evidence across a compact set of resampling scales and mirrored views. MCR first resamples the input at multiple scales that are rounded to integer multiples of the model’s patch grid to avoid misalignment artifacts. Thereafter,MCR performs forward inference (optionally via an overlap-aware sliding window) and up-samples logits back to the original image resolution. In each scale,MCR computes a horizontally mirrored view,maps its outputs back to the canonical coordinate frame,and aggregates the mirrored and canonical predictions. A final cross-scale averaging yields the comprehensive dense prediction. The overall design explicitly preserves the zero-shot characteristics of the underlying VLM while using only forward-time operations to improve spatial precision.

Result

We evaluate TG-CLIP on eight public segmentation benchmarks that represent diverse settings with and without explicit background classes. These benchmarks include PASCAL VOC,PASCAL-Context,COCO-Object,ADE20K,COCO-Stuff,Cityscapes,and commonly used VOC subsets. In feature extraction,we use the CLIP ViT-B/16 encoder in a frozen configuration. All experiments are performed without any additional training or pixel-level fine-tuning. The hyperparameters are kept consistent across datasets. Across the evaluated benchmarks,TG-CLIP achieves an average mIoU of 45.5%,outperforming several recent baselines. In particular,TG-CLIP exceeds the second-ranked ProxyCLIP (40.1%) by 4.4 percentage points in averaged mIoU. Qualitative inspections indicate that TG-CLIP yields crisp boundaries,optimal preservation of thin structures,and reduced spurious activations in complex scenes. Detailed ablation studies confirm that both operators contribute to performance gains: TGR primarily improves local category separability and boundary fidelity,while MCR increases robustness to scale and viewpoint variability. Sensitivity analyses show that TGR is robust to a wide range of temperature and injection-strength settings,and MCR exhibits a measurable accuracy-efficiency tradeoff dictated by the number of scales and whether mirrored views are included. We report and recommend practical operating points that balance latency and accuracy for real-world deployment. TG-CLIP demonstrates that carefully designed inference-stage interventions can substantially close the gap between region-optimized VLM representations and the needs of dense,pixel-level segmentation without retraining. Given that all modifications occur at inference time,our framework is particularly attractive for engineering-constrained scenarios,such as embedded perception or rapid prototyping,where re-training is costly or infeasible. Nevertheless,several limitations remain: the multi-scale and mirrored inference steps introduce additional computation proportional to the number of scales and views; performance on extremely small or heavily occluded instances is still constrained by the spatial granularity of the frozen encoder; and our method does not eliminate the potential gains from task-specific supervised fine-tuning when labels are available. Finally,although our experiments include multiple standard benchmarks,further validation on real-world robotic perception tasks and diverse domain shifts can effectively elucidate generalization properties.

Conclusion

We propose TG-CLIP,a training-free inference methodology that combines text-guided recalibration with multi-scale flipped fusion to enhance pixel-level semantic discrimination and robustness of VLM-based open-vocabulary segmentation. TG-CLIP is simple to implement,preserves the zero-shot strengths of pretrained models,and yields consistent empirical gains across diverse benchmarks. We release implementation details and deployment recommendations to facilitate adoption and further research on inference-centric approaches for dense vision-language tasks. The codes are available at:https://www.scidb.cn/s/7baeYf.

open-vocabulary semantic segmentation (OVSS)  /  zero-shot learning  /  vision-language models (VLM)  /  contrastive language-image pre-training (CLIP)  /  pixel-level semantic segmentation  /  training-free inference
王馨静, 高颖, 邹亚琦, 朱政宇, 徐春雪, 赵琦. 语义对齐与局部感知驱动的开放词汇语义分割. 中国图象图形学报, 2026 , 31 (8) : 2774 -2784 . DOI: 10.11834/jig.250540
Xinjing Wang, Ying Gao, Yaqi Zou, Zhengyu Zhu, Chunxue Xu, Qi Zhao. Semantic alignment and locality-driven open-vocabulary semantic segmentation[J]. Journal of Image and Graphics, 2026 , 31 (8) : 2774 -2784 . DOI: 10.11834/jig.250540
  • National Natural Science Foundation of China(62401310);Shandong Province University Youth Innovation Technology Support Program(2024KJG053)
Year 2026 volume 31 Issue 8
18
9
Cite this Article
BibTeX
Article Info
doi: 10.11834/jig.250540
  • Receive Date:2025-11-05
  • Online Date:2026-10-30
  • Published:2026-08-16
Article Data
Affiliations
History
  • Received:2025-11-05
  • Revised:2026-01-16
Funding
National Natural Science Foundation of China(62401310);Shandong Province University Youth Innovation Technology Support Program(2024KJG053)
Affiliations
    School of Data Science,Qingdao University of Science and Technology,Qingdao 266000,China
References
Share
https://castjournals.cast.org.cn/joweb/zgtxtxxb/EN/10.11834/jig.250540
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT