Power system states are not accurately known ahead of real time and are subject to natural variability, which leads to uncertainty in the system. Sources of uncertainty are amongst others load, renewable energy sources and contingencies. An aggregate measure for system's uncertainty is determined in order to have a single quantified value of the uncertainty level in the system. This allows to compare performance of power system reliability management according to various reliability criteria at different uncertainty levels in a transparent manner. A case study for a 5 node test system illustrates the use of the aggregate uncertainty measure and the impact of different uncertainty levels on short term reliability management of transmission system operators according to three reliability criteria. Results of the case study illustrate that probabilistic reliability management can lead to significant improvements in performance in terms of total system cost and curtailment of non-flexible load compared to reliability management based on deterministic N-0 and N-1 criteria, especially at high uncertainty levels in heavily loaded system conditions.