As asymmetric interior permanent magnet synchronous motor (AIPMSM) has excellent performance but complicated topological structure, a novel high-resolution encoding and edge smoothing method is proposed for topology optimization of the asymmetric rotor of interior permanent magnet synchronous motor (IPMSM) in this study. This method aims to solve complex electromagnetic design problems with time-dependent performance through a multi-objective genetic algorithm (MOGA) integrated with a high-resolution encoding and edge smoothing method. The complex structure is represented by a high-resolution image-like matrix and then vectorized by the edge smoothing method. Therefore, the commonly used discrete binary encoded variables related to the finite element (FE) model are replaced with a vectorized topological structure and other control variables. In this sense, high-resolution matrix and edge smoothing methods are used for the first time to represent the rotor topology of AIPMSMs. Compared with the traditional topology optimization method, the proposed method has the advantage of expressing more complex and vectorized topological structures; meanwhile, the obtained performance is accurate and trustworthy using conventional FE simulation. Numerical results show that a stable convergence is achieved with the avoidance of checkerboards and material overlapping. It is shown that the proposed method can find solutions with better performances, in comparison with the reference model.