Back-up heaters are essential for sustainable retrofit heat pump systems to achieve low capital costs and high system temperatures. Despite its importance, current literature focuses primarily on single aspects of the interaction between the back-up heater and the heat pump system. Furthermore, influences of varying scenarios are typically not considered. This paper simultaneously investigates the impact of 18 different scenarios on the optimal answer to the questions: Which back-up heater to choose, where to place it, and how to control it? A scenario consists of boundary conditions for weather, building envelope, radiator sizing, operational envelope, and the electricity-to-gas price/emission ratio, respectively. Using annual dynamic Modelica simulations, we evaluate and assess all interdependencies based on a full factorial design. We analyze final energy consumption, thermal comfort, and back-up heater as objectives. For gas-fired back-up heaters, the optimal placement and control align with current state-of-the-art recommendations. However, for electric back-up heaters, current guideline recommendations yield up to 30% higher operational costs and emissions compared to our findings. Consequently, future studies should develop optimal design rules for sustainable retrofit heat pump systems.