An open world object detection method based on shape perception and class balance optimization was proposed to address the issue of poor prediction performance of unknown class objects in open world object detection. Unknown classes referred to classes that were not labeled during the training phase. Due to the lack of guidance from labels, detecting unknown class objects was a challenging task. An unknown class enhanced detector has been constructed as an unknown class detection branch. During training, this detector was supervised using only known class labels, allowing it to learn the similarities in features of known class objects and generalize to unknown class objects. To improve the detector's sensitivity to unknown classes, the region proposal network (RPN) module's ability to distinguish between foreground and background was utilized. A specific filtering method was employed to select results with “unknown class potential” from the RPN output, which were then used as pseudo labels in the training process. Due to the absence of confidence scores, traditional non-maximum suppression (NMS) methods were difficult to apply for post-processing unknown objects. Therefore, a redundant unknown object suppression mechanism was designed, consisting of a center point-based grouping strategy and a redundancy score matrix based on shape perception. The center point-based grouping strategy included three methods based on the unknown class center points to determine the suppression range. Subsequently, a redundancy score matrix was constructed based on the redundancy scores of each prediction box within the group to suppress highly redundant predictions. Experimental results on open world object detection datasets demonstrated that the open world object detection based on shape perception and class balance optimization maintained high recall rates for unknown classes while achieving high prediction accuracy. This method effectively addressed the challenges of open world scenarios and avoided generating a large number of useless predictions.