Agricultural management methods, such as cultivation or fallowing, have led to significant changes in soil fertility and hence, crop yield. Such changes may have stemmed from changes in soil microbial communities and associated biogeochemical processes. This phenomenon is particularly true in organic-poor soil in the Loess Plateau of China. In this study, we examined three existing soil management regimes as part of a 10-year field experiment and evaluated their effects on fungal and bacterial community structures by performing high-throughput 454 pyrosequencing. These management regimes were (i) fertilized winter wheat (Triticum aestivum L.) (FW), (ii) continuous natural fallow with weeds but without crop grown (NF), and (iii) continuous bare fallow without weeds or crop grown (BF). After 10 years, soil organic carbon (SOC), microbial biomass carbon (MBC), and available potassium (K) concentrations were highest in NF. Soil N behaved differently, with BF obtaining the highest nitrate nitrogen (N). Meanwhile, slight differences in total N (TN) were observed among FW, NF, and BF. Available phosphorus (P) was highest and available K was lowest in FW. Microbial communities were dominated by Ascomycota (59.1% of fungal sequences), and Acidobacteria, Actinobacteria, Bacteroidetes, Firmicutes, and Proteobacteria (75.7% of bacterial sequences) in FW, NF and BF at the phylum level. Soil management regimes did not affect the fungal and bacterial richness and diversity but significantly modified their community compositions. Compared with FW, the abundances of Ascomycota (fungi phylum) and Alternaria, Gibberella, and Emericella (fungi genus) were increased by NF, whereas the values of Chaetomium, Humicola, and Cryptococcus (fungi genus) were decreased by BF. The abundances of Verrucomicrobia (bacteria phylum), and Steroidobacter (bacteria genus) were increased by NF, and Bacteroides (bacteria genus) was increased by BF. Canonical correspondence analysis showed that SOC, available P, and TN might be the key factors in community formation. Therefore, the decadal absence of plants (BF) affected soil fertility by increased available K and nitrate N, whileas natural fallow (NF) affected soil fertility by increased SOC, available K, and MBC, and they all changed fungal and bacterial community compositions.