This study proposes a novel modified grey-wolf optimization algorithm (MGWOA) to enhance power system stability. The power system stabilizer and static synchronous series compensator (SSSC) are used as damping controllers. Additionally, fractional-order PID (FOPID) controller is used to handle the system nonlinearities and thus achieve better performance. The control parameters are tuned using the proposed MGWOA method which has been verified on unimodal and multimodal functions. Single-machine infinite bus (SMIB) and multimachine power system (MMPS) are taken as case studies to analyze the efficacy of the proposed controller. Minimization of rotor speed deviation is considered an objective function. The results obtained from the MGWOA-tuned FOPID-based damping controllers are compared with those obtained using recently developed efficient and competitive heuristic algorithms. It was observed that the MGWOA method is well-suited for damping low-frequency oscillations. Furthermore, statistical analysis is performed on the obtained results to justify the superiority of the MGWOA method. The simulation results suggest that the MGWOA exhibits superior performance characteristics when applied to a real power system.