It has been broadly reported that the production of animal-derived foods significantly contributes to the environmental footprint of the agri-food sector, considering, among others, such indicators as land use, greenhouse gas emissions, and the water footprint. However, the environmental sustainability aspects of whole diets (i.e., more plant-based vs. meat-containing) have been studied so far to a limited extent, in many cases not taking into consideration various regional settings, which to a great extent determine meat- and other animal-derived foods’ substitutes consumed by vegetarians and vegans. The aim of this study was therefore to assess the environmental indicators of vegetarian, vegan, and meat-containing diets of a selected group of Polish consumers. Based on three-day food records of 24 respondents and the published data on the environmental footprint of a range of foodstuffs, the three dietary environmental indicators were calculated. In addition, the results were standardized in terms of dietary protein intake and energy value. The study showed the elimination of meat and other animal-derived foods from the respondents’ diet was predominantly motivated by their concerns related to animal welfare issues, which appeared to be a stronger factor than the willingness to reduce the diets’ environmental footprint. Following the results standardization, the studied vegetarian and vegan diets were characterized by 47.0% and 64.4% lower carbon footprint, 32.2% and 60.9% lower land use indicators, and 37.1% and 62.9% lower water footprints, respectively, compared to the meat-containing diet. Animal-derived foods, including milk and dairy, appeared to be the main contributors to all three environmental footprint indicators of both the meat-containing and the vegetarian diets. In the vegan group, the environmental footprint was found to be mainly influenced by the consumption of legumes and legume-based foods, cereal products, potatoes, sugar, products containing cocoa and vegetables, with nuts showing especially significant contribution to the fresh water consumption. The study confirms moving towards more plant-based diet has a potential to significantly reduce the diet’s environmental footprint. It also contributes to creating a ‘roadmap’ for consumers, to encourage them to plan their diets responsibly, taking into consideration both the health and the environmental sustainability aspects.