Intelligent-based protective relays in the power system is currently growing very rapidly with the use of embedded systems and artificial intelligence algorithms. Overcurrent relays operated based on the standard characteristic curve will move toward digital overcurrent relay (DOCR) using a flexible characteristic curve based on the user settings. DOCR with the standard characteristic curve cannot be used if the user must make modifications to the curve for field implementation purpose. In this paper, a new DOCR based on modeling a characteristic curve using Artificial Neural Network (ANN) will be presented. Four types DOCR with the standard and user-defined characteristic curve were developed and tested. Results demonstrate their successful operation with highly accurate results. The results also prove that the proposed DOCR with user-defined characteristic curve may serve as a convenient alternative to the conventional DOCR particularly on the implementation phase.