Minimizing the energy offset between the lowest exciton and charge-transfer (CT) states is a widely employed strategy to suppress the energy loss (Eg/q - VOC) in polymer:non-fullerene acceptor (NFA) organic solar cells (OSCs). In this work, transient absorption spectroscopy is employed to determine CT state lifetimes in a series of low energy loss polymer:NFA blends. The CT state lifetime is observed to show an inverse energy gap law dependence and decreases as the energy loss is reduced. This behavior is assigned to increased mixing/hybridization between these CT states and shorter-lived singlet excitons of the lower gap component as the energy offset ΔECT-S1 is reduced. This study highlights how achieving longer exciton and CT state lifetimes has the potential for further enhancement of OSC efficiencies.