收藏切换
A dual-modal salient object detection network for 360° omnidirectional images
收藏切换
Xiaolei Chen*, Zhihua Zhong, Yujie Shen
Journal of Image and Graphics | 2026, 31(10) : 3618 - 3635
Less
收藏切换
Journal of Image and Graphics | 2026, 31(10): 3618-3635
• Image Analysis and Recognition •
A dual-modal salient object detection network for 360° omnidirectional images
Full
Xiaolei Chen*, Zhihua Zhong, Yujie Shen
Affiliations
  • School of Microelectronics Industry-Education Integration,Lanzhou University of Technology,Lanzhou 730050,China
Published: 2026-10-16 doi: 10.11834/jig.250564
Outline
收藏切换
Objective

Salient object detection (SOD) seeks to emulate human visual attention by locating and segmenting the most conspicuous objects in visual scenes. While deep-learning-based 2D RGB and RGB-D SOD methods have advanced substantially,360° omnidirectional (panoramic) images introduce unique challenges,most notably spherical-to-plane projection distortions (e.g.,equal rectangular projection(ERP) artifacts),severe polar-region deformation,and boundary discontinuities across the left-right wraparound. At the same time,depth cues are known to improve geometric reasoning in conventional RGB-D SOD,but their effective exploitation for 360° SOD is underexplored due to modality-alignment difficulties,noise in estimated depth,and the lack of fusion architectures tailored to panoramic projection effects. The objective of this work is to investigate and demonstrate how carefully designed cross-modal modeling and projection-aware feature processing can harness depth information to substantially improve salient object detection on ERP panoramic images,while remaining robust to depth estimation noise and projection artifacts.

Method

We propose an asymmetric dual-branch U-Net architecture for RGB-D 360° SOD that explicitly accounts for ERP projection properties and for cross-modal alignment. The model contains three novel components: 1) a panoramic-aware receptive field module (PA-RFM) that enhances direction-sensitive context modeling via a longitude-latitude-global triplet attention scheme and a ring-padding/cropping strategy to respect horizontal periodicity; 2) an attention-guided fusion module (AFM) that performs adaptive,dynamic weighting of RGB and depth channels using combined channel and spatial attention plus a lightweight modality weight generator to suppress noisy depth responses while preserving boundary cues; and 3) a cross-modal guided collaborative decoder in which a fusion-centric decoding path is explicitly guided by an independent high-resolution RGB decoding path via attentional gating,yielding superior edge and texture recovery. The RGB encoder is a pretrained SAM2 Hiera backbone with lightweight adapters to limit finetuning cost; the depth encoder is a ResNet-34 modified for single-channel input. PA-RFM employs a four-branch multiscale dilated design with a panoramic context aggregation module (PCAM) at each branch to merge longitude attention (via circular padding + horizontal convolutions),latitude attention (via latitude-weighted vertical convolutions that attenuate polar exaggeration),and global semantic attention (via latitude-weighted global average pooling (GAP) and channel recalibration). Training is performed in PyTorch on NVIDIA RTX 3090 hardware using AdamW (lr = 1 × 10-3,weight decay = 5 × 10-4),cosine annealing to 1 × 10-7,batch size 4,up to 50 epochs,input resolution 512 × 1 024,and a structured loss composed of weighted BCE + weighted IoU with multilayer deep supervision.

Result

We evaluate the proposed model on two public panoramic SOD benchmarks augmented with depth maps produced by a state-of-the-art 360° depth estimator: 360-SOD (500 ERP images; 400 train/100 test) and 360-SSOD (1 105 ERP images; 850 train/255 test). Quantitatively,our method achieves MAE = 0.015 1,max-F = 0.838 8,mean-F = 0.815 0,max-E = 0.933 1,mean-E = 0.924 0,and Sm = 0.873 6 on 360-SOD (same scores on 360-SSOD),outperforming 10 representative RGB-D 2D SOD methods and seven representative RGB 360° SOD methods in objective metrics and visual quality. Relative to the strongest competing baseline,our mean absolute error(MAE) is reduced by 13.7%,max-F increases by 4.64%,mean-F by 3.24%,and Sm by 1.96%,demonstrating substantial gains particularly in boundary accuracy and structural consistency. Extensive qualitative comparisons show our model is markedly better at detecting objects in severely distorted polar regions,maintaining left-right boundary continuity,and recovering fine edges in cluttered or low-contrast scenes. Ablation studies confirm the contribution of each module: PA-RFM yields the largest single-module improvement (vs. RFB and a no-PA-RFM baseline),PCAM’s three attention branches (longitude,latitude,global) act synergistically,and a ring-padding width of k = 2 provides the best tradeoff between boundary continuity and redundancy. AFM outperforms simple fusion strategies (add,concat,multiply) and several state-of-the-art (SOTA) fusion blocks (MobileSal,HIDANet,CPNet variants). The cross-modal guided decoder meaningfully improves edge/detail recovery over a variant that omits the RGB decoding path. We also test robustness to different depth estimators and find consistent improvements over RGB-only baselines; performance varies modestly with depth source,with Joint_360depth yielding the best overall results but not causing catastrophic degradation when replaced.

Conclusion

This study demonstrates that depth information can be effectively and robustly exploited for 360° omnidirectional SOD when combined with projection-aware receptive field design and modality-aware fusion and decoding strategies. The proposed PA-RFM addresses ERP-specific distortions (horizontal periodicity and polar exaggeration) via circular padding,latitude weighting and global recalibration; AFM adaptively reconciles noisy depth with RGB structure; and the cross-modal guided collaborative decoder preserves high-resolution RGB priors to restore fine boundaries and textures lost in fused representations. Together,these innovations lead to consistent and remarkable gains across standard panoramic SOD benchmarks,especially in polar regions and at projection boundaries. The approach is general and can be extended to other panoramic vision tasks and to setups with sensor-captured depth; future work will explore domain adaptation to real depth sensors,multiprojection fusion (ERP + CMP),and model compression for mobile/AR/VR deployment.

salient object detection(SOD)  /  360° omnidirectional image  /  RGB-D  /  cross-modal fusion  /  attention mechanism
陈晓雷, 钟智华, 申玉杰. 双模态360度全景图像显著目标检测网络. 中国图象图形学报, 2026 , 31 (10) : 3618 -3635 . DOI: 10.11834/jig.250564
Xiaolei Chen, Zhihua Zhong, Yujie Shen. A dual-modal salient object detection network for 360° omnidirectional images[J]. Journal of Image and Graphics, 2026 , 31 (10) : 3618 -3635 . DOI: 10.11834/jig.250564
  • National Natural Science Foundation of China (61967012)
Year 2026 volume 31 Issue 10
28
14
Cite this Article
BibTeX
Article Info
doi: 10.11834/jig.250564
  • Receive Date:2025-11-07
  • Online Date:2026-10-30
  • Published:2026-10-16
Article Data
Affiliations
History
  • Received:2025-11-07
  • Revised:2026-01-21
Funding
National Natural Science Foundation of China (61967012)
Affiliations
    School of Microelectronics Industry-Education Integration,Lanzhou University of Technology,Lanzhou 730050,China
References
Share
https://castjournals.cast.org.cn/joweb/zgtxtxxb/EN/10.11834/jig.250564
Share to
QR

Scan QR to access full text

Cite this article
BibTeX
Citations
表12种不同金属材料的力学参数
科
Family
属数
Number of
genus
种数
Number of
species
占总种数比例
Percentage of
total species (%)
属
Genus
种数
Number of
species
占总种数比例
Percentage of total
species (%)
鹅膏菌科Amanitaceae 2 11 5.26 鹅膏菌属 Amanita 10 4.78
小菇科 Mycenaceae 2 12 5.74 丝盖伞属 Inocybe 5 2.39
多孔菌科 Polyporaceae 8 14 6.70 蜡蘑属 Laccaria 5 2.39
红菇科 Russulaceae 3 23 11.00 小皮伞属 Marasmius 6 2.87
小菇属 Mycena 11 5.26
光柄菇属 Pluteus 5 2.39
红菇属 Russula 17 8.13
栓菌属 Trametes 5 2.39
关闭全屏
  • BibTeX
  • EndNote
  • RefWorks
  • TxT