Climate change can significantly impact agriculture, leading to food security challenges. Most previous studies have investigated the direct climate impact on crops while neglecting the impact of heat stress on agricultural labor. Here, we assess the economic consequences of climate impacts on four major crops—maize, soybean, wheat, and rice—for scenarios involving low and high greenhouse gas emissions. Our analysis is based on the output from a new generation of global climate and crop models to drive a multiregional economic model. We find that, even under a high-emission scenario, the effect of CO$_2$ fertilization could lead to higher yields, resulting in lower prices for major crops, except for maize. However, heat-induced losses in agricultural labor could offset the potential economic benefits of CO$_2$ fertilization in crop production in Asia and Africa. Our findings emphasize the importance of addressing heat-stress impacts on agricultural labor through proactive adaptation measures.