At present, the International Maritime Organization (IMO) has issued the final guidelines of the direct stability assessment of surf-riding/broaching for the second generation intact stability criteria, and how to accurately and efficiently predict the surf-riding/broaching is a key problem to be solved for the direct stability assessment of surf-riding/broaching. So a surge-sway-heave-roll-pitch-yaw coupled mathematical model (6-DOF) is established in this paper. Firstly, the heave and pitch motions are considered in the surge-sway-roll-yaw maneuvering mathematical model, and the amplitudes and phases of heave and pitch motions are calculated by a strip method using an enhanced integrating method, which can solve the problem of divergence resulting from direct seakeeping calculation in time-domain for high speed vessel in stern-quartering waves. Secondly, the Froude-Krylov forces and diffraction forces are calculated by integrating the wave pressure up to the mean wave surface. At the same time, nonlinear hydrodynamic derivations, heel-induced hydrodynamic forces and nonlinear roll damping are considered in the mathematical model. The hydrodynamic lift forces due to the coexistence of wave particle velocity and ship forward velocity are taken into account in the propeller thrust and rudder force model. And the real-time emersion of double rudders in waves is considered in the rudder force model. Finally, a tumblehome ship with twin propellers and double rudders is utilized to study surf-riding/broaching in stern-quartering waves, and the effect of initial relative position of the ship to waves on predicting surf-riding/broaching motion is investigated. The computation results show that the established 6-DOF mathematical model has enough accuracy to be used for the direct stability assessment of the surf-riding/broaching failure mode.