Research on crop diseases has become a hot topic of the application of artificial intelligence technology in smart agriculture. Existing studies are restricted by the factors such as difficulties in collecting data, high cost of technology implementation, and complex crop disease trends. Plant electronic medical records (PEMRs) formed by Beijing Plant Clinic provides a new idea for the diagnosis and prevention of crop diseases and pests. PEMRs are stored in the form of multi-modal data, containing a wealth of plant information, disease information, and environmental information. Therefore, how to mine PEMRs information and utilize it to assist follow-up research is an urgent problem to be solved. In view of the information representation ability of knowledge map, the mining ability of machine learning and the feature extraction ability of deep learning, structured data is used to construct the knowledge graph of crop diseases and pests, unstructured data and domain knowledge are used for knowledge enhancement according to the characteristics of PEMRs. Further, Neo4j graph database and graph data science (GDS) combined with machine learning algorithm was employed to conduct association mining from three dimensions of popular point discovery, link discovery and similar disease discovery. At the same time, text feature extraction and disease diagnosis were realized by using unstructured text data based on the bidirectional encoder representation from transformers (BERT) with convolutional neural network (CNN) model, and intelligent service was realized by simulating plant doctor. The comprehensive accuracy of 20 common diseases was up to 93.13%. This study can provide theoretical support for timely diagnosis, symptomatic control, scientific medication guide and assistant decision-making of crop diseases and pests, and innovate a new model and new business of social service of agricultural science and technology.