Penetration level of renewable energy (RE) in the utility grid is continuously increasing to minimize the environmental concerns, risk of energy security, and depletion of fossil fuels. The uncertain nature and availability of RE power for a short duration have created problems related to the protection, grid security, power reliability, and power quality. Further, integration of RE sources near the load centers has also pronounced the protection issues, such as false tripping, delayed tripping, etc. Hence, this paper introduces a hybrid grid protection scheme (HGPS) for the protection of the grid with RE integration. This combines the merits of the Stockwell Transform, Hilbert Transform, and Alienation Coefficient to improve performance of the protection scheme. The Stockwell Transform-based Median and Summation Index (SMSI) utilizing current signals, Hilbert Transform-based derivative index (HDI) utilizing voltage signals, and Alienation Coefficient index (ACI) utilizing voltage signals were used to compute a proposed Stockwell Transform-, Hilbert Transform-, and Alienation-based fault index (SAHFI). This SAHFI was used to recognize the fault conditions. The fault conditions were categorized using the number of faulty phases and the proposed Stockwell Transform and Hilbert Transform-based ground fault index (SHGFI) utilizing zero sequence currents. The fault conditions, such as phase and ground (PGF), any two phases (TPF), any two phases and ground (TPGF), all three phases (ATPF), and all three phases and ground (ATPGF), were recognized effectively, using the proposed SAHFI. The proposed method has the following merits: performance is least affected by the noise, it is effective in recognizing fault conditions in minimum time, and it is also effective in recognizing the fault conditions in different scenarios of the grid. Performance of the proposed approach was found to be superior compared to the discrete wavelet transform (DWT)-based method reported in the literature. The study was performed using the hybrid grid test system realized by integrating wind and solar photovoltaic (PV) plants to the IEEE-13 nodes network in MATLAB software.