Environmental factors that are important in shaping microbe community structure are less explored along elevation in the alpine grassland ecosystem of Tibet Plateau, which is generally phosphorus limited. Here, we examined soil prokaryote communities at three elevations to explore soil prokaryote community distribution and mediation factors in Noijin Kangsang Peak, Tibetan Plateau. Results showed prokaryote community compositions differed significantly by elevations. Topsoil or subsoil prokaryote richness and Shannon diversity were significantly lower at the middle than other elevations, while significantly higher aboveground biomass (AGB) and available P (AP) were found at the middle elevation. The importance of P for both soil layers was discovered by variation partitioning analysis based on redundancy analysis, finding that soil AP and total phosphorus, interacted with pH, explained 43% the variance in topsoil prokaryote community compositions, while soil AP, as well as AGB, explained 44% in subsoil. Consistently, structural equation model also revealed that AP was a mediating factor for prokaryote community diversity. Other than plant beta diversity, soil prokaryote beta diversity positively correlated with AP difference significantly. Taken together, the distribution patterns of soil prokaryote community were distinct along elevations even in a small scale in Noijin Kangsang Peak and was likely mediated predominantly by soil AP in both topsoil and subsoil.