Humans are an increasingly dominant driver of Earth's biological communities, but differentiating human impacts from natural drivers of ecosystem state is crucial. Herbivorous fish play a key role in maintaining coral dominance on coral reefs, and are widely affected by human activities, principally fishing. We assess the relative importance of human and biophysical (habitat and oceanographic) drivers on the biomass of five herbivorous functional groups among 33 islands in the central and western Pacific Ocean. Human impacts were clear for some, but not all, herbivore groups. Biomass of browsers, large excavators, and of all herbivores combined declined rapidly with increasing human population density, whereas grazers, scrapers, and detritivores displayed no relationship. Sea-surface temperature had significant but opposing effects on the biomass of detritivores (positive) and browsers (negative). Similarly, the biomass of scrapers, grazers, and detritivores correlated with habitat structural complexity; however, relationships were group specific. Finally, the biomass of browsers and large excavators was related to island geomorphology, both peaking on low-lying islands and atolls. The substantial variability in herbivore populations explained by natural biophysical drivers highlights the need for locally appropriate management targets on coral reefs.