China’s agriculture is increasingly becoming more specialized. However, specialized production has disrupted traditional farming culture and may threaten sustainable mental health. This study takes Chinese farmers’ subjective happiness and agricultural production outsourcing as the research object, in an attempt to reveal the possible unhappy impacts of Chinese-style agricultural specialization represented by agricultural production outsourcing. First, we construct a theoretical framework of the relationship between agricultural production outsourcing and farmers’ subjective well-being. Secondly, based on more than 3800 household survey data collected by the Chinese Academy of Social Sciences in 2020, we use the classical econometrics and psychological analysis methods such as the Ordered Probit model and the instrumental variable estimation to conduct a rigorous impact assessment. The results show that for every doubling of agricultural outsourcing expenditure, the probability that farmers think they are very happy decreases by about 21%, and the probability that they think they are relatively happy decreases by about 9%. The groups affected by the negative psychological impact mainly include farmers growing rice and corn, farmers in hills and mountains, and farmers with small-scale operations. Further analysis shows that outsourcing risks, the weakening of farmers’ professional autonomy, and family split caused by agricultural outsourcing bring unhappiness, and the increase in income cannot offset the negative psychological effect of outsourcing. The findings of this study may bring inspiration to other countries with agricultural outsourcing markets and programs to improve the national subjective well-being.