Abstract Tripping time and optimal overcurrent coordination are nowadays one of the main power system protection concern, due to the high penetration of renewable energy sources in the electrical power network. In this article, optimal coordination scheme has been developed using nonstandard tripping characteristics for the Over Current Relays (OCRs) in a power network connected to a Photovoltaic System (PV). Considering the impact of Distribution Generation (DG) on fault currents and locations on the network, this paper presents and verifies an optimal coordination scheme based on two optimization methods, Particle Swarm Optimization (PSO) algorithm and Genetic Algorithm (GA). The optimal coordination scheme aims to improve the sensitivity and reliability of the protection system by using a nonstandard tripping characteristic to reduce the operating time of OCRs. A specific case study, IEEE-9 bus connected to PV system is developed and presented using Industrial software (ETAP) and the results of the proposed optimal coordination scheme is compared to conventional characteristics in the literature. The proposed nonstandard OCR coordination scheme and optimization algorithms are evaluated using different fault and power network model scenarios. The results indicate that the proposed optimal nonstandard scheme successfully reduce the total operating time of OCRs and improve the relay sensitivity during minimum and maximum fault scenarios.