In arid to semi-arid regions, vulnerability to climate change combined with the overexploitation of water resources is jeopardizing food security. In the Souss-Massa region in central Morocco, the rural population relies on growing olives for a living. The management of these orchards is mostly traditional under rainfed irrigation, which induces a high level of dependence on climate variability. In the present study, we investigate the long-term trends of the relationship between the observed olive yields and global climate patterns during the period 1973–2014. We apply lagged Spearman’s correlations and cross-wavelet analysis to detect the potential influence of El Niño-southern oscillation (ENSO), the Indian Ocean Dipole (IOD), North Atlantic oscillation (NAO) and Pacific decadal oscillation (PDO) on the yield variability of olive orchards. The results of a Mann-Kendall test show a statistically significant decreasing trend in olive yields during the studied period. Statistically significant negative correlations were observed for (lag = −1) with spring and summer NINO 3.4 and with summer and autumn PDO. No statistically significant correlations between olive yields and NAO and IOD were observed. The results of wavelet coherence between annual olive yields and PDO and ENSO revealed that the highest values of power spectrum coherence occurred during the (lag = 0) spring PDO and (lag = −1) spring ENSO, both with an antiphase relationship. During the studied period, the extreme events of El Niña and El Niño years corresponded to below average yields.