The intermittence and volatility of photovoltaic power generation threaten the stability of the power grid, and high-precision prediction can provide key support for power system scheduling. Solar irradiance prediction methods based on meteorological data and sky images have been widely studied, but the mining of multi-scale temporal features and sky image features is still very limited, and the correlation between features is not fully utilized. Therefore, this paper proposes an ultra-short-term solar irradiance prediction model based on multi-source data and Transformer framework. First, the long-term and short-term memory neural network-Transformer is used to learn the long-term and short-term temporal variation characteristics of multivariate meteorological sequences. Secondly, the convolution visual Transformer is used to model the global information and local structure of the sky image, and the deep image features are extracted. Finally, a cross-modal feature fusion module based on attention mechanism is proposed, which uses image features to represent temporal features, realizes cross-modal feature fusion, and outputs prediction results by decoder. Experimental results show that the proposed method has better prediction performance than the state-of-the-art methods. Compared with the case of only using meteorological data, the NRMSE of the model is reduced by 7.4% on average. The model significantly improves the prediction accuracy by mining the features of multi-source data (meteorological time series and sky images) and using the cross-modal attention mechanism to effectively mine their correlation.