[Objective] Colony extraction and counting is essential in agriculture, food, and health industries. Currently, most of the available algorithms for automatic counting of colonies use colony culture dishes and has poor applicability to colony count plates. In addition, the current technologies have good performance in conventional segmentation of adherent objects, while their accuracy remains to be improved for the segmentation and counting of adherent colonies due to the unique morphological characteristics of colonies. [Methods] To solve such problems, we proposed a colony segmentation and counting algorithm based on target color base and gradient direction matching. Firstly, the color feature of the colony in the image was used as a base to convert the image into a base space to enhance the difference between the colony and the background. Secondly, the gradient magnitude feature of the colony image was used to filter the gradient direction, and then the matching was performed through the gradient direction, thereby segmenting the adherent colonies. Finally, non-maximum suppression was employed to screen and count the colonies. [Results] Through experiments, the counting accuracy of the algorithm in this study reaches 98.00%, demonstrating its capability to meet practical requirements. [Conclusion] In the context of targeted segmentation and counting of colonies, the algorithm studied in this paper not only exhibits high counting accuracy but also demonstrates good robustness. This algorithm had not only high counting accuracy but also good robustness, producing excellent results in the colony segmentation and counting of colony count plates from different manufacturers. However, it showed decreased counting accuracy in the detection and segmentation of large-area targets. Therefore, this algorithm is suitable for the detection and segmentation of small targets such as colonies.