Abstract The energy retrofitting of existing buildings reduces the energy use in the operation phase but the use of additional materials influence the energy use in other life cycle phases of retrofitted buildings. In this study, we analyse the life cycle primary energy implications of different material alternatives when retrofitting an existing building to meet high energy performance levels. We design retrofitting options assuming the highest and lowest value of final energy use, respectively, for passive house standards applicable in Sweden. The retrofitting options include the thermal improvement of the building envelope. We calculate the primary energy use in the operation phase (operation primary energy), as well as in production, maintenance and end-of-life phases (non-operation primary energy). Our results show that the non-operation primary energy use can vary significantly depending on the choice of materials for thermal insulation, cladding systems and windows. Although the operation energy use decreases by 63–78%, we find that the non-operation energy for building retrofitting accounts for up to 21% of the operation energy saving, depending on the passive house performance level and the material alternative. A careful selection of building materials can reduce the non-operation primary energy by up to 40%, especially when using wood-based materials.