Understanding the spatial variability of CO2 emissions among and within cities and its driving factors is a prerequisite for emission reductions. Using 10 × 10 km national grid emission data (circa 2007), we quantified the spatial distribution of total CO2 emissions and emission efficiency of Chinese cities at the prefectural city scale. The emission efficiency was measured according to the emission intensity, CO2 emissions per capita, residential CO2 emissions per capita and industrial CO2 emissions per unit area. We found substantial variability in the total CO2 emissions among cities ranging from <0.1 to 214.3 million tons. High total CO2 emissions were mostly concentrated in northern, eastern and northeastern China. An overall analysis of the total CO2 emissions and emission intensity revealed that 75% of the cities in northern China had higher total emissions and lower efficiencies than the national average, and these cities shall be the main targeted cities for CO2 emissions reduction and emission intensity improvement. Additionally, urban districts had higher total CO2 emissions and emissions per capita than their surrounding regions for the majority of the prefectural cities. Four indicators, including the industrial structure, gross domestic product (GDP), total population, and size of the built-up area, were significantly related to the CO2 emissions and emission efficiency. Industrial structure was the most important driving force. Our results underscore the need to design region- and/or city-specific reduction strategies instead of a one-size-fits-all policy, and provide strategic information to public and private decision makers on controlling total emissions and improving emission efficiency.