The electric power system (EPS) is a large, multi-parameter, non-linear and dynamic system. The problem of calculation of relay protection (RP) settings has become more urgent nowadays. The situation is exacerbated by the active implementation of renewable energy sources, FACTS, etc., which significantly change the “traditional” EPS and their operating modes. The problem can be solved by deep analysis of functioning of main elements of RP devices in the specific operating conditions and revision of coefficients used in settings calculation. That can be done using RPs detailed mathematical models and modern EPS simulators. The results of the analysis will make it possible to formulate a new methodology for setting up RP. That is a final goal. In the framework of solving this problem, the novel approach for developing RPs’ detailed mathematical models is formulated and theoretically proved. On the basis of this approach, mathematical models of the system “instrumental transformer - auxiliary transformer - analog filter” (measuring part of digital transformer differential protection) for different types of auxiliary current transformers (active and passive) and filters (Butterworth, Chebyshev, Bessel) are developed. A comparative numerical analysis of their frequency and phase responses is carried out, including taking into account the magnetization of instrumental current transformers. Summarizing, the theoretical and practical studies presented in the article allows formulating requirements for RPs’ detailed mathematical models, which will be used in the further research.