Cyclin-dependent kinase 5 (CDK5), a serine/threonine kinase, is one of the non-typical members of the CDKs family. CDK5 is mainly activated by non-cyclin activators p35 or p39 (as well as their respective fragments p25 and p29) to phosphorylate downstream substrates and regulate numerous neural and non-neural functions. Increasing evidence has confirmed that the overactivation of CDK5/p25 complex is closely related to neurodegenerative diseases, cancers, diabetes and inflammation. Consequently, CDK5 has become an important target in multiple diseases treatment. Nevertheless, to date, no selective CDK5 inhibitors are currently in the clinical stage. On the other hand, pan-CDK inhibitors are limited in clinical trials, due to their poor clinical efficacy and toxic side effects caused by the extensive inhibition of other kinases. In view of this, selective CDK5 inhibitors are of great significance not only for elucidating its exact biological functions, but also exploring the possibility of CDK5 inhibitors as a safe and effective therapy. This paper provides a brief overview of the structure and function of CDK5 protein as well as its relationship with diseases. In addition, the structural types and binding modes of CDK5 inhibitors targeting ATP active sites are also highlighted. Finally, we summarize and prospect the strategies to improve the selectivity of CDK5 inhibitors.