Abstract This paper proposes a classification of reliability criteria for power systems based on four characteristics: (i) the set of considered system states, (ii) the objective function, (iii) the allowed real-time actions and (iv) optional non-technical constraints. Because selecting a reliability criterion involves a trade-off between multiple opposing objectives, this paper suggests the use of five performance indicators to evaluate the performance of reliability criteria: (i) expected total cost, (ii) service reliability level, (iii) inequality between consumers in terms of service reliability, (iv) availability, integrity and confidentiality of input information and (v) ease-of-use. Case studies for a five-node and 24-node test system illustrate the multi-dimensional analysis of six reliability criteria. These criteria range from the deterministic N − 1 criterion to an unconstrained probabilistic criterion that aims to minimize expected total cost. The analysis finds that moving from deterministic to probabilistic reliability criteria decreases expected total cost. The largest savings of expected total cost are due to a trade-off between preventive and corrective actions. A smaller portion of savings is due to the inclusion of load curtailment actions in the trade-off on top of preventive and corrective control. Limits on individual or aggregate service unreliability levels decrease service unreliability, but increase expected total costs when compared to an unconstrained probabilistic approach.