The costs of unintended side effects of agriculture such as water pollution cannot be directly observed in markets. However, the values society places on healthy agricultural environments are increasingly reflected in payments to farmers for measures to avoid or reduce environmental damage. This paper presents a framework for estimating external costs of agriculture from payment rates of agri-environment measures addressing specific externality issues. The framework is applied to the broad range of agri-environment measures implemented in Swiss agricultural policy. Estimates of external costs are derived for emissions of greenhouse gases, ammonia, nitrate and pesticides, soil erosion, habitat deficits, and animal suffering. The total external costs of Swiss agriculture are estimated at CHF 3.651 billion (CHF 3494 per hectare) when the calculations are based on the agri-environment measures’ average avoidance costs and of CHF 5.560 billion (CHF 5321 per hectare) when the calculations are based on highest observed avoidance costs. Potential applications include internalization policies, evaluations of agri-environment support, and integrated environmental and economic accounting.