Separating the effect of resource competition from allelopathy in plants is challenging and it has never been attempted in closely related co-occurring bryophytes. In peatlands, peat mosses (Sphagnum spp.) show niche differentiation along water table level (WTL) gradient. Our aim was to evaluate whether the hummock species, S. magellanicum would be a winner at low WTL due to its allelopathic advantage and the hollow species, S. angustifolium would win by virtue of its superior competitive ability but not of allelopathy at high WTL due to dilution of its allelochemicals. We used a nested, field experimental design, with two WTL treatments—low WTL (hummock habitat) and high WTL (hollow habitat)—and three different inter-specific interactions: 1) monoculture; 2) mixed culture without activated charcoal; and 3) mixed culture with activated charcoal added to the neighbor. We measured growth and biochemical traits of the two species and compared the index of relative neighbor effect on each other. We discovered a trade-off between biomass production (competitive outcome) and phenolic content (allelopathy) in these species. At low WTL, allelopathy of the hummock species is the main mechanism to suppress the hollow species, whereas at high WTL, competition is the main driver to suppress the hummock species. Competitive advantage in Sphagnum is mediated by both resource competition and allelopathy of the co-occurring species through niche differentiation along a WTL gradient. Unlike vascular plants, Sphagnum mosses can serve as excellent model organisms in studying allelopathic interaction since they bypass the complexity of plant-soil interactions.