SummaryArbuscular mycorrhizal fungi (AMF) play an important role in maintaining plant diversity and productivity in grassland ecosystems. However, very few studies have investigated how AMF and plant communities co‐vary between contrasting environments in natural ecosystems. Intensive sampling (50 soil samples) was conducted in natural open grasslands at both 3570 and 4556 m on Mount Segrila on the Southeast Tibetan Plateau. We used 454‐pyrosequencing to investigate soil AMF communities and to explore relationships between AMF diversity and plant richness, productivity and community composition. AMF diversity was negatively correlated with plant richness at 3570 m but positively at 4556 m. Differences in AMF community composition between elevations were attributable to plant community composition, soil pH and available phosphorus concentration. The AMF community was more phylogenetically clustered at the higher elevation than the lower elevation. However, greater phylogenetic clustering (under dispersion) of AMF communities at the two elevations was positively correlated with above‐ground biomass. Our results indicate that plant community composition and environmental filtering are the primary drivers structuring the AMF community. Phylogenetic relatedness may be important in explaining the function of AMF communities in alpine ecosystems.