In the busy waters of nearshore port channels, physical Aids to Navigation (AtoN) are highly susceptible to collisions, which may cause structural damage and prevent them from operating normally. To address the research gap in autonomous navigation of AtoN, including the inability of existing methods to achieve autonomous navigation and collision avoidance and their insufficient integration with artificial intelligence technologies for intelligent navigation, this paper proposes an autonomous navigation method for AtoN based on improved Proximal Policy Optimization (PPO) combined with the Velocity Obstacle (VO) algorithm. The PPO algorithm was improved by introducing an offline learning strategy to enhance network update efficiency. By designing the autonomous navigation action space for AtoN, the model can achieve faster convergence. However, the model is prone to falling into local optima during convergence. To avoid this problem, the VO algorithm is incorporated to represent the state space, and a reward function is designed using velocity vectors. On the basis of calculating the distance-based reward function, an additional reward function based on position and velocity information is introduced to improve convergence speed and enable AtoN to adaptively select the optimal velocity in nearshore port channels. To further enhance algorithm robustness, an entropy reward term is added to the PPO algorithm to encourage greater exploration. Experimental results show that, compared with the method without the entropy reward term, the proposed method improves the average navigation step length and average navigation velocity by 9.3% and 10.2%, respectively, while maintaining a model test success rate of over 99%. The proposed method enables a safe operational mode featuring unmanned control, autonomous navigation, and collision avoidance for AtoN. It can reduce the collision damage risk of AtoN in busy waters and lower the costs of manual inspection and maintenance. The study is of significance for accelerating the construction of a modern maritime navigation support system and ensuring maritime navigation safety.