The need for achieving efficient and sustainable use of water resources is pressing, however, this often requires better understanding of the potential of water conservation, taking into account the impact on return flows, and the costs in relation to sectoral benefits. Using modelling and limited observational data we explore the costs and potential water savings of 24 combinations of water conservation measures in the Rufiji basin, Tanzania. We compare these costs with estimates of the value such water savings could generate from water use in three important economic sectors; agriculture, energy and downstream ecosystems with high tourism potential. The cost of water conservation measures (median: 0.07 USD m−3) is found to be: higher than the value of most uses of water for agriculture (growing crops in expanded irrigation sites) and the median value for hydropower generation (from a new mega dam currently under construction); and lower than the ecosystem value. Nevertheless, under our modelling assumptions, the volume of additional water required to supply planned irrigation expansion in the basin could be reduced by 1.5 BCM using water conservation methods that would be financially viable, given the value of competing uses of water. Water savings of this magnitude would reduce potential trade-offs between use of water for hydropower and ecosystem services, by allowing peak environmental flow releases even in dry years, and without reducing firm energy generation. This methodology is transferable and relevant for producing realistic assessments of the financial incentives for long-term sustainable water use in agriculture, given incentives for other uses. With most reservoirs now being built for multiple purposes improved understanding of trade-offs between different sectors and functions is needed.