Variations in the climate constitute a significant threat to the productivity of food crops in the Gambia. A good understanding of the influence of climate variability on crop production is vital for climate resilience and improved food security. This study examined the trends, relationships, and the extent to which growing season temperatures and the SPEI (Standardized Precipitation and Evapotranspiration Index) impacted sorghum, millet, maize, and rice yields in three agro-ecological regions of the Gambia during 1990–2019. Mean temperatures and the SPEI exhibited increasing trends while observed yields showed a decline across all regions. The SPEI had a significant positive relationship with yields, and temperatures were negatively associated with yields. Though yield response to climate variability differs among regions, 20% to 62% of variations in the four crop yields were due to climate trends. The combined effect of the SPEI and temperatures decreased yields from 3.6 kg ha−1 year−1 to 29.4 kg ha−1 year−1, with the most severe decline observed in rice and maize yields in the Sahelian zone. Although uncertainties might arise from not considering related extreme climate events, this study highlights how past climate trends affect cereal yields in the Gambia; thus, any unfavorable change in the local climate could have severe repercussions on the country’s food security. There is a need for concerted efforts to increase investments in adaptation strategies to lessen the effects of the climate for improved crop productivity.