Existing deep learning-based modulation recognition methods are difficult to adapt to HF channels with significant time-varying characteristics, limiting their application in modulation recognition for High Frequency (HF) signals. In addition, in non-cooperative HF communication scenarios, the training dataset is often difficult to cover all possible modulation types, so that open-set modulation recognition also has important practical significance. This paper proposes a multi-modal feature fusion-based open-set modulation recognition method by combining communication domain knowledge and open-set recognition techniques,which effectively reduces the impact of time-varying HF channels and unknown modulation types on recognition performance. The proposed method first utilizes communication domain knowledge to obtain multimodal features that are robust to channel variations,and then extracts discriminative deep feature representations through multimodal feature fusion and deep feature learning to effectively identify known and unknown modulation types. In addition, the method also generates dummy samples through manifold mixing strategy to assist network training, which can enhance the network's ability to identify unknown types. Experimental results indicate that the proposed method outperforms existing open-set modulation recognition methods. When the channel conditions of training and testing signals are the same, the proposed method improves by over 3% in open-set recognition performance. When the channel condition of testing signals is drastically changed, that is, the channel conditions of the training and testing signals are different,the proposed method improves by over 8% compared to existing method, which exhibits strong robustness to channel variations.