Analyzing the safety of the dam is of vital importance for maintaining their safe operation. Relying on the high asphalt concrete core rockfill dam project of the lower reservoir of a planned pumped storage power station, the mechanical characteristics tests of rockfill materials in transition layer, upstream rockfill area and downstream rockfill area were carried out. The Duncan E-B model and rheological model parameters of rockfill materials were measured. On this basis, the three-dimensional nonlinear stress-deformation finite element calculation of the dam was carried out. The safety of the dam was demonstrated from the aspects of dam body, core wall and cutoff wall. The results show that under the normal water level, the extreme settlement value of the dam is -22.29 cm and the settlement rate is 0.28%. The major and minor principal stresses of the dam are 1.38 MPa and 0.54 MPa, respectively. The extreme values of core wall displacement along the river, dam axial displacement and vertical displacement are 11.55 cm, -3.70 cm/3.63 cm and -21.46 cm, respectively. The extreme value of principal tensile strain is -0.8%. The extreme displacement values of the cutoff wall along the river, the axial direction and the vertical direction of the dam are 1.32 cm, -0.11 cm/0.02 cm and -0.54 cm, respectively. The extreme values of major and minor principal stresses are 6.82 MPa and -1.20 MPa respectively. After rheology and wetting are taken into account, the extreme value of dam settlement is -35.00 cm, and the settlement rate is 0.44%. The extreme values of core displacement along the river, dam axial and vertical displacement are 13.65 cm, -4.97 cm/4.77 cm and -33.01 cm, respectively. The comprehensive analysis results indicate that the distribution and extreme value of stress and deformation of the dam are reasonable, and the tensile strain of asphalt concrete core wall and the tensile stress of concrete wall in dam foundation are not beyond the standard value, so it can be judged that the dam can meet the requirements of safe operation.