Three structurally identical polymers, except for the number of fluorine substitutions (0, 1, or 2) on the repeat unit (BnDT-DTBT), are investigated in detail, to further understand the impact of these fluorine atoms on open circuit voltage (V(oc)), short circuit current (J(sc)), and fill factor (FF) of related solar cells. While the enhanced V(oc) can be ascribed to a lower HOMO level of the polymer by adding more fluorine substituents, the improvement in J(sc) and FF are likely due to suppressed charge recombination. While the reduced bimolecular recombination with raising fluorine concentration is confirmed by variable light intensity studies, a plausibly suppressed geminate recombination is implied by the significantly increased change of dipole moment between the ground and excited states (Δμ(ge)) for these polymers as the number of fluorine substituents increases. Moreover, the 2F polymer (PBnDT-DTffBT) exhibits significantly more scattering in the in-plane lamellar stacking and out-of-plane π-π stacking directions, observed with GIWAXS. This indicates that the addition of fluorine leads to a more face-on polymer crystallite orientation with respect to the substrate, which could contribute to the suppressed charge recombination. R-SoXS also reveals that PBnDT-DTffBT has larger and purer polymer/fullerene domains. The higher domain purity is correlated with an observed decrease in PCBM miscibility in polymer, which drops from 21% (PBnDT-DTBT) to 12% (PBnDT-DTffBT). The disclosed "fluorine" impact not only explains the efficiency increase from 4% of PBnDT-DTBT (0F) to 7% with PBnDT-DTffBT (2F) but also suggests fluorine substitution should be generally considered in the future design of new polymers.