Recognition of reef-building corals is important for protecting and monitoring coral reef ecosystems. Deep learning, as an advanced technology in image recognition, has been increasingly applied in coral recognition. However, its performance is still challenged by several issues, such as the imbalance of samples among different coral categories within a dataset and the limitation of data diversity. The former makes the deep learning model more likely to extract features from classes with a large number of samples and, therefore, decreases its ability to recognize small-sample-size corals, which often refer to endangered ones needing to be protected. The latter further reduces the performance of deep learning in recognizing corals with different appearances and are captured in variant environments. To solve these two problems, this study develops a reef-building coral recognition method by integrating a category-quantity adaptive deep data augmentation algorithm and transfer learning. To address the first problem, a category-quantity adaptive deep data augmentation algorithm named DeepSMOTE-F₁ is proposed. This algorithm improves the existing DeepSMOTE by introducing a sample-size determination stagey using an F1-score based evaluation metric. It can adaptively augment the number of samples of each category of corals according to its recognition performance so that the deep learning model can fully learn features from each class of corals. For the second problem, transfer learning is used to further enhance the model's ability to extract features. The experimental results on three widely used public coral recognition datasets, RSMAS, EILAT, and EILAT2 show that the recognition accuracy of the proposed DeepSMOTE-F₁ is improved by 2.88%, 0.39%, and 1.54%, respectively, compared with the traditional DeepSMOTE; and the accuracy of the integrated method is improved by 0.76%, 1.40% and 1.30% compared with the existing deep learning methods for coral recognition.