Rural tourism is one of the important ways to achieve rural revitalization and one of the choices for farmers to improve their viability. Therefore, the socio-ecological systems (SES) analysis framework and obstacle factor analysis can provide a new perspective for the evaluation of farmers’ behavior adaptation processes. In order to test the adaptation mechanisms of farmers’ lives in rural tourism destinations, we applied this analysis framework to Ying-Tao-Gou Village in China. The study found that the livelihood adaptation strategies of local farmers can be divided into the categories of “tourism livelihood”, “part-time livelihood”, “worker livelihood”, and “farming livelihood”. The livelihood adaptation of local farmers presented a relatively balanced picture. There were significant differences in the livelihood adaptability of different types of farmers, but tourism farmers had the strongest livelihood adaptability. There were various barriers to the livelihood adaptation of different types of farmers, such as household savings and education level. Finally, to improve the adaptability of farmers’ livelihoods, amendments can be made in policy formulation, tourism development, and by the farmers themselves. This study is of great significance for the government in formulating policies to improve people’s livelihoods, to stimulate the vitality of rural development, and to accelerate the development of rural revitalization.