In order to realize high-precision localization of Autonomous Valet Parking (AVP) system, this paper proposed a multi-sensor fusion localization algorithm based on vector map. In order to achieve long-term consistency of the localization effect, this paper employed visual semantic features to express the environment around the autonomous vehicle, and matched it with the vectorized map items to solve the absolute pose based on visual input. To achieve the above functions, this paper proposed a novel matching strategy to address the inconsistency of expression between observational data and vectorized map data. Meanwhile, by designing a reasonable error function and modelling the pose estimation problem as a nonlinear optimization problem, the method can achieve high-precision solution results. In addition, in order to improve the robustness of the localization results, this paper employed the Kalman filter based on the error state to fuse the results of the visual matching localization with the measured values of the IMU and wheel speedometer, and realized a tightly coupled modular localization method. The verification results based on real data show that the algorithm proposed in this paper is feasible in both theory and practice, and it can be seen from the comparison test with the mainstream methods that this method can achieve higher performance.