Recent advances in organic solar cells based on non-fullerene acceptors (NFAs) come with reduced non-radiative voltage losses (ΔVnr). Here we show that, in contrast to the energy-gap-law dependence observed in conventional donor:fullerene blends, the ΔVnr values in state-of-the-art donor:NFA organic solar cells show no correlation with the energies of charge-transfer electronic states at donor:acceptor interfaces. By combining temperature-dependent electroluminescence experiments and dynamic vibronic simulations, we provide a unified description of ΔVnr for both fullerene- and NFA-based devices. We highlight the critical role that the thermal population of local exciton states plays in low-ΔVnr systems. An important finding is that the photoluminescence yield of the pristine materials defines the lower limit of ΔVnr. We also demonstrate that the reduction in ΔVnr (for example, <0.2 V) can be obtained without sacrificing charge generation efficiency. Our work suggests designing donor and acceptor materials with high luminescence efficiency and complementary optical absorption bands extending into the near-infrared region. Organic solar cells based on non-fullerene acceptors have enabled high efficiencies yet their charge dynamics and its impact on the photovoltaic parameters are not fully understood. Now, Chen et al. provide a general description of non-radiative voltage losses in both fullerene and non-fullerene solar cells.