The deep-sea represents a substantial portion of the biosphere and has a major influence on carbon cycling and global biogeochemistry. Benthic deep-sea prokaryotes have crucial roles in this ecosystem, with their recycling of organic matter from the photic zone. Despite this, little is known about the large-scale distribution of prokaryotes in the surface deep-sea sediments. To assess the influence of environmental and trophic variables on the large-scale distribution of prokaryotes, we investigated the prokaryotic assemblage composition (Bacteria to Archaea and Euryarchaeota to Crenarchaeota ratio) and activity in the surface deep-sea sediments of the Mediterranean Sea and the adjacent North Atlantic Ocean. Prokaryotic abundance and biomass did not vary significantly across the Mediterranean Sea; however, there were depth-related trends in all areas. The abundance of prokaryotes was positively correlated with the sedimentary concentration of protein, an indicator of the quality and bioavailability of organic matter. Moving eastwards, the Bacteria contribution to the total prokaryotes decreased, which appears to be linked to the more oligotrophic conditions of the Eastern Mediterranean basins. Despite the increased importance of Archaea, the contributions of Crenarchaeota Marine Group I to the total pool was relatively constant across the investigated stations, with the exception of Matapan-Vavilov Deep, in which Euryarchaeota Marine Group II dominated. Overall, our data suggest that deeper areas of the Mediterranean Sea share more similar communities with each other than with shallower sites. Freshness and quality of sedimentary organic matter were identified through Generalized Additive Model analysis as the major factors for describing the variation in the prokaryotic community structure and activity in the surface deep-sea sediments. Longitude was also important in explaining the observed variability, which suggests that the overlying water masses might have a critical role in shaping the benthic communities.