From time to time a sequence of unexpected and overlapping contingencies may lead to power system angular instability and even blackouts if not addressed adequately by means of an out-of-step (OOS) protection system. The motivation of the paper is an attempt to develop a workable prototype of the OOS protection system. The deficiencies of the protection currently used in the Latvian Power System network are highlighted and a new protection structure is proposed. The protection system comprises of several strategically located terminals, exchanging information in real time by means of a communication network. The OOS condition detection method is based on system-wide generation sources, electromotive forces, vectors, and angle control. The network splitting decision is based on generator coherence evaluation. Protection terminals determine online the groups of coherent generators and choose the splitting boundary from a predefined transmission lines (TLs) cut sets list. The protection system structure, algorithm of operation, and possible IEC 61850 communication standard-based implementation are described.