Crashworthiness optimization is an effective way to achieve better passive safety protection performance of vehicles，but current optimization focuses on improving numerical response，while neglecting the control of a category response，namely，deformation modes. The deformation mode of key components is related to the effectiveness of vehicle force transmission path design. If an unsatisfactory deformation mode occurs in the optimization solution，the effectiveness of the optimization result cannot be guaranteed. Therefore，in this study a machine learning based deformation mode control optimization method is proposed to improve the crashworthiness index while ensuring that all samples in the optimization solution deform in ideal modes. Structural deformation is represented in the form of images，and deep learning auto encoder is used to extract deformation features and cluster them to identify different deformation modes. Then，machine learning prediction models based on Light Gradient Boosting Machine （LightGBM） are established for the identified deformation modes and numerical responses. Finally，the optimization is solved based on the machine learning prediction models. The proposed machine learning optimization method is validated using a full vehicle frontal collision case，and the results show that while improving the numerical crashworthiness responses，the deformation mode of the longitudinal beam is ensured to deform in an ideal mode. This study demonstrates the prospects of machine learning in improving the effectiveness of structural optimization.