Analysis of the change of overall energy efficiency performance of an energy system is a fundamental work for the energy-saving policymaking. However, previous studies seldom focus on energy stages from useful energy to final service, while most attention are paid on stages from energy source to useful energy. In this paper, we develop a high-resolution the Societal Exergy Analysis and Logarithmic Mean Divisia Index (SEA-LMDI) method to analysis changes and driving factors of the aggregate exergy efficiency, in which the boundary of the SEA is extended to passive systems and final services, and a LMDI decomposition method is developed to quantify contributions of efficiency factors and structure factors of all six stages on the aggregate exergy efficiency. A case study of China during 2005–2015 reveals that: (a) the aggregate exergy efficiency from energy source to final service is only from 3.7% to 4.8% during 2005–2015, showing a huge theoretical potential of efficiency improvement. (b) Large passive losses are identified in passive systems and nearly 2/3 of useful energy can be theoretically saved by improving passive systems. (c) Deep analysis of industrial coal-fired boilers indicates that the internal structural adjustments are also important for the aggregate improvement.