Abstract The high geothermal heat exchanger (GHE) installation cost is the main challenge encountered in the widespread use of geothermal heat pump (GHP) systems, so its optimization is vital for reducing the costs. In the present study, five main parameters ─the radius, length, and the number of wells, the external pipe's radius, and the flow discharge inside the pipe─ are optimized by genetic algorithm (GA) for a residential building in Tehran. Moreover, sensitivity analysis of several design parameters, which are not considered in the objective function, indicates that pipe thermal conductivity, borehole thermal conductivity, soil thermal conductivity, and borehole distance parameters had the highest effects on entropy generation (EG), respectively. Therefore, this approach can help engineers to select the most efficient parameters for improving their design. Eventually, the optimized GHP is investigated by energy and economic viewpoints. One-year energy simulation of these systems is conducted to determine the energy consumption. Simulation results suggest that the annual energy consumption of the GHP with the coefficient of performance (COP) of 5.6 is 10.111 MWh; whereas, the annual consumption of heat pumps with an air-source heat pump equals 42.222 MWh, which is 4.17 times greater than that of the GHP. A simulation over a 10-year period is also performed to consider the drop in performance of the GHE over time. Furthermore, the economic analysis results suggest that the payback period of this system is about 7.4 years, and the energy subsidy paid by the government will be reduced annually to 14, 417, 839 Iranian Rial (IRR).