To stabilize frequency in a power system, this research study suggests a novel modified Gorilla Troops Optimizer (mGTO) technique, which builds on the original technique, and offers notable gains in effectiveness and efficiency when solving real-world optimization problems. A thorough comparative analysis reveals that the mGTO algorithm is the best option, outperforming its counterparts in terms of stability and overall performance. Interestingly, mGTO performs better than any of its competitors in terms of stability, making it the best option. The mGTO algorithm significantly reduces implementation time and enhances solution quality compared to the conventional GTO algorithm. A new linearized Phillips–Heffron model with an IPFC was developed to investigate power systems’ stability. To effectively dampen low-frequency oscillations, an auxiliary controller for modeling the IPFC is proposed. It provides four options for damping controllers, and the recommended mGTO algorithm is used to adjust the controller parameters. This method is superior to traditional controllers in stability control and has undergone extensive validation. It is a crucial instrument for controlling the frequency of an SMIB power system based on IPFC. Based on the simulation results, the updated strategy that has been suggested is the most effective way to define the mentioned damping controller by considering the percentage improvement in the goal function value.