Farmland abandonment, including perennial and seasonal abandonment, is an important process of land use change that matters most to food security. Although there is a great deal of studies on farmland abandonment, seasonal abandonment, which is as serious as perennial abandonment, has attracted little academic attention. This paper takes Hunan Province in central China as its study area and uses a spatial regression model to examine the driving factors of seasonal farmland abandonment at the county level. Our results show that farmland abandonment has striking spatial relativity, and there are two clustering zones with a high index of farmland abandonment (IFA) in the Dongting plain and the basin in south-central Hunan, while a clustering zone of low IFA can be found in the mountains of southwest Hunan. Farmland abandonment at the regional level is negatively affected by the land productive potentialities, proportion of mechanized planting, ratio of effective irrigation, and distance to provincial capital, while it is positively associated with the variables mountainous terrain, per capita farmland area, and labor shortage. Additionally, farmland abandonment is also affected by adjacent areas through its spatial dependence. In short, seasonal farmland abandonment is also driven integratedly by the socioeconomic and environmental dimensions and spatial interaction of farm abandonment.