Aiming at the problems that the large-scale pre-training language model faces when dealing with news headlines, such as huge parameters, inefficient use of contextual semantic features and circular convolution neural network’s neglect of the importance of initial input elements, a news headline classification method that combines ERNIE(enhanced representation through knowledge integration) of mixture-of-expert model and recurrent convolution neural network with attention mechanism were proposed. Firstly, the text was encoded with the help of MoE’s improved ERNIE technology, and then the text was classified with attention RCNN (recurrent convolutional neural networks)on the basis of preserving the word order and characteristics of the text. In order to improve the classification ability, RCNN was improved by calculating the input fusion context weight. In the process of calculating the weights of experts in MoE, Gumbel-Softmax was selected as a new gating function to improve the traditional Softmax function, so as to better control the smoothness. According to the experimental results, it is found that compared with the traditional classification methods, the classification method proposed in this study shows significant advantages and greatly reduces the number of parameters. On this basis, the F₁ value is increased by 0.51% compared with the traditional model. After the ablation experiment, the feasibility of this classification method in the classification task has been confirmed.