Since total factor productivity growth plays an essential role in China’s economic growth, the source of this growth has been a critical issue over the past decades. Hence, this paper applies an input slack-based productivity (ISP) index to investigate the contributors (i.e., labor and capital inputs) to China’s total factor productivity growth. The ISP index, combining the features of the directional distance function and Luenberger productivity index, can calculate the productivity change of each input factor under the total factor framework. According to the decomposition analyses, we find that China is confronting a dual challenge in total factor productivity growth: first, capital productivity growth exhibits a remarkable slowdown after the mid-1990s; second, although labor productivity continually expands, the relative labor efficiency among provinces has deteriorated since the 2000s. The results imply that the government should not only advocate upgrading industrial structure, but also consider balanced regional development policies for China’s sustainable growth.