Human food production is the largest cause of global environmental changes. Environmental benefits could be achieved by replacing diets with a high amount of animal-sourced foods with more plant-based foods, due to their smaller environmental impacts. The objective of this study was to assess the environmental impacts of the three most common plant-based milk alternatives (PBMAs)—oat, soy, and almond drink—in comparison with conventional and organic cow milk. Life cycle assessments (LCA) were calculated by the ReCiPe 2016 midpoint method, in addition to the single issue methods “Ecosystem damage potential” and “Water scarcity index”. PBMAs achieved lower impact values in almost all 12 of the calculated impact categories, with oat drink and the organic soy drink being the most environmentally friendly. However, when LCA results were expressed per energy and by the protein content of the beverages, the ranking of the beverages, in terms of their environmental impacts, changed greatly, and the results of PBMAs approached those of milk, particularly with regard to the protein index. The study highlights the importance of considering a broader range of impact categories when comparing the impacts of PBMAs and milk.