The majority of global fish stocks lack adequate data for their stock statuses to be assessed using conventional stock assessment methods. Data-limited methods, such as CMSY, have been increasingly recommended as new solutions for stock assessment and fishery management. However, CMSY is highly dependent on data quality, and the reliability of the method is yet to be verified under circumstances of limited length of time series data and variable observational errors. In this study, we investigated effects of lengths of catch time series, stages of fishery development, and levels of observational errors in catches on stock assessment of three economically-important species in the Yellow Sea using CMSY method. The results show that chub mackerel (Scomber japonicus), hairtail (Trichiurus lepturus), and silver pomfret (Pampus argenteus), all have been overfished (B/B_MSY<1 and F/F_MSY>1), with their yields higher than estimated MSY since 2000, and although their fishing intensities have been reduced over the most recent decade, their biomasses remain at low levels (B/B_MSY<1). The retrospective analysis show small differences in the results of stock assessment for the three species, indicating that the assessments are robust enough with long time series data. As to effects of lengths of catch time series, the assessments are more stable using time series data covering a period of both rise and fall in catches. The effect of observational errors in catches is also tested, showing that when the error is >20%, the model tend to overestimate MSY and B_MSY, but the assessment remains robust enough. This study suggests that cautions should be undertaken in the application of CMSY by using longer time series of catch data and, in the presence of high uncertainty in the assessment, more conservative measures should be taken in fishery management.