Water issue is one of the challenges of urban sustainability in developing countries. To address the conflict between urban water use and economic development, it is required to better understand the decoupling states between them and the driving forces behind these decoupling states. The transformed Tapio decoupling model is applied in this paper to study the decoupling relationship between urban industrial water consumption and economic growth in Beijing and Shanghai, two megacities in China, in 2003–2016. The factors driving decoupling are divided into industrial structure effect, industrial water utilization intensity effect, economic development level effect, and population size effect through Logarithmic Mean Divisia Index (LMDI) method. The results show that: (1) the decoupling states of total water consumption and economic growth in Beijing and Shanghai are mainly strong decoupling and weak decoupling. In comparison, Shanghai’s decoupling effect is better than Beijing; (2) regarding decoupling elasticity, Beijing is higher than that of Shanghai in tertiary industry and lower in primary industry and secondary industry. As a result, Beijing’s decoupling level is worse than Shanghai in tertiary industry, while better in primary industry and secondary industry; (3) The common factors that drive the two megacities’ decoupling are industrial structure effect and industrial water utilization intensity effect. The effects of economic development level and population size mainly present weak decoupling in two megacities, but the decoupling state is optimized year by year. Finally, based on the results, some suggestions for achieving the sustainable development of urban water use are proposed.