Objective To analyze the value of tumor growth pattern (TGP) and tumor budding (TB) on judgment of the prognosis of patients with stage Ⅰ-Ⅲ colorectal cancer. Methods The clinical data of 152 patients with stage Ⅰ-Ⅲ colorectal cancer, admitted and undergone radical operation from December 2012 to June 2020 in Hainan Hospital of Chinese PLA General Hospital, were collected and retrospectively analyzed. TGP and TB were interpreted with HE staining and patients were divided into TGP-A (dilatant growth type and intermediate type) or B (invasive growth type), TB-low or TB-high subgroups. The differences of clinicopathological features among these subgroups were analyzed; Kaplan-Meier analysis was performed to analyze the disease-free survival (DFS) and overall survival (OS) of these subgroups. Finally, Cox proportional risk model was used to analyze the risk factors influencing prognosis of colorectal cancer patients. Results Statistical differences of tumor differentiation degree and N stage existed in TGP-A or TGP-B, TB-low or TB-high subgroups (P<0.01); in addition, the characteristics were more obvious of high CEA level (P=0.03), T₃+T₄ stage (P<0.01) and TNM Ⅲ stage (P<0.01) in TGP-B groups. Between TGP-A and TGP-B subgroups, and between TB-low and TB-high subgroups, the statistical significantly difference existed in DFS (TGP: Log-rank=10.06, P<0.01; TB: Log-rank=6.62, P=0.01) and OS (TGP: Log-rank=6.53, P=0.01; TB: Log-rank=4.90, P=0.03). TGP was an independent risk factor for both DFS (HR=2.95, 95%CI 1.54-5.63, P<0.01) and OS (HR=2.37, 95%CI 1.14-4.94, P=0.02). Conclusion Both TGP and TB were more reliable prognostic markers for patients with stage Ⅰ-Ⅲ colorectal cancer, the ones presented TGP with infiltrative pattern or TB ≥10/0.785 mm² would have poor prognosis.