The coordination of optimal overcurrent relays (OCRs) for modern power networks is nowadays one of the critical concerns due to the increase in the use of renewable energy sources. Modern grids connected to inverter-based distributed generations (IDGs) and synchronous distributed generations (SDGs) have a direct impact on fault currents and locations and then on the protection system. In this paper, a new optimal OCR coordination scheme has been developed based on the nonstandard time–current characteristics (NSTCC) approach. The proposed scheme can effectively minimize the impact of distributed generations (DGs) on OCR coordination by using two optimization techniques: genetic algorithm (GA) and hybrid gravitational search algorithm–sequential quadratic programming (GSA–SQP) algorithm. In addition, the proposed optimal OCR coordination scheme has successfully employed a new constraint reduction method for eliminating the considerable number of constraints in the coordination and tripping time formula by using only one variable dynamic coefficient. The proposed protection scheme has been applied in IEEE 9-bus and IEC MG systems as benchmark radial networks as well as IEEE 30-bus systems as meshed structures. The results of the proposed optimal OCR coordination scheme have been compared to standard and nonstandard characteristics reported in the literature. The results showed a significant improvement in terms of the protection system sensitivity and reliability by minimizing the operating time (OT) of OCRs and demonstrating the effectiveness of the proposed method throughout minimum and maximum fault modes.