Urban public services can significantly promote the improvement and capacity of urban innovation, yet there are differences. Although some studies have analyzed the impact of public services on technological innovation, few studies have taken Chinese cities at the prefecture level and above as samples to build an urban public service evaluation index system and focus on analyzing the differences and impact mechanisms. Based on panel data of 277 cities at the prefecture level and above from 2005 to 2019, this paper empirically studies the differences and mechanisms of the impact of urban public services on urban innovation. The results show that urban public services can significantly promote the improvement of urban innovation ability. After endogenous treatment and robustness testing, the empirical results remain robust. There are differences in the impact of urban public services on urban innovation: the positive effect of basic public services on urban innovation is greater than that livelihood public services. The innovation effect of public services in small and large cities is significant, but not in medium-sized cities. The effect of urban public services in eastern and central regions on improving innovation capability is higher than that in the western and northeastern regions. In cities with different levels of administration, the technological innovation effect of public services in key cities is not significant, while in generalized cities, it is significant. The study also found that public services can positively affect urban innovation through the talent agglomeration effect, industrial structure effect, and income scale effect.