Revealing the effect of particle size distribution (PSD) to the stiffness characteristics of the red stratum soil-rock mixture (RS S-RM) has significant implications for the subgrade construction, with Sichuan Basin as a representative. 20 large-scale triaxial tests with different PSDs were conducted. Introducing the concept of effective dominant skeleton size ( d_{\mathrm{e}\mathrm{d}}), digital image processing technology was utilized to collect the d_{\mathrm{e}\mathrm{d}}. The relative dominance ratio ( D_{\mathrm{d}}) describing the structural skeleton changes after the test was defined. The response of the stiffness characteristics of the RS S-RM regarding confining pressure ( {\sigma }_3) and D_{\mathrm{d}} were investigated. Validation experiment and related research verify the credibility of the formation mechanism of stiffness. The results indicate that as the shearing process, more and more particles near the shear plane transition from their initial interlocking state to sliding friction. Consequently, the soil-stone framework evolves from a suspended-dense structure to a skeleton-dense structure, and finally transforms into a skeleton-void structure; the initial deformation modulus ( E_0) relationship with {\sigma }_3 and D_{\mathrm{d}} follows: E_0 = 14.50 + ( 1 + 36.33 {{\sigma }_3}^{0.1} ) {\mathrm{e}}^{1.06D_{\mathrm{d}}}. The relationship between the tangent deformation modulus ( E_{\mathrm{t}}) of the specimen at failure {\epsilon }_1 = 15 \mathrm{\%} is E_{\mathrm{t}} = - 6.06 + ( 1 + 0.08 {\sigma }_3 ) {\mathrm{e}}^{-0.09D_{\mathrm{d}}}. Validation experiment and related research confirmed the response of E_0 and E_{\mathrm{t}} to {\sigma }_3 and D_{\mathrm{d}}. The stiffness characteristics of the RS S-RM can be predicted solely based on the PSD and {\sigma }_3.