The search for alternatives to the CdS buffer layer in Cu(In,Ga)Se $_2$ (CIGS) solar cells has turned out to be quite promising in terms of power conversion efficiency. In this paper, the typically used chemical-bath-deposited CdS layer is compared with an atomic-layer-deposited Zn $_{1-x}$ Sn $_{x}$ O $_{y}$ (ZnSnO). An optical study by external quantum efficiency and photoluminescence on the influence of different buffer layers on the defect properties of CIGS is presented. For both buffer layers, the CIGS bulk and CIGS/buffer interface are strongly influenced by electrostatic fluctuating potentials, which are less pronounced for the sample with the ZnSnO buffer layer. This is associated with a lower concentration of donor defects at the CIGS near-interface layer. A change in the bandgap of the CIGS as a consequence of the buffer layer deposition is observed. This study expands the knowledge of defects in the complex quaternary semiconductor CIGS, which, as discussed, can be affected even by the choice of buffer layer and its deposition process.