Abstract Spectral beam splitting technology based on optical nanofluids (NFs) is a promising field of solar energy research. However, the challenges to optimize the optical properties and enhance the high temperature stability of NFs are always exist. In this study, Indium Tin Oxide (ITO) nanoparticles were selected as the research object because of the excellent optical properties. The Oleylamine-capped ITO nanoparticles were prepared by chemical reduction. A novel NF based on Therminol 66 was successfully prepared. The high temperature stability and optical properties of obtained Oil-based NFs were analyzed by the gravitational sedimentation method and spectrophotometry. Moreover, we experimentally studied the effect of single reaction condition on the morphology and Sn-doping of particles. The results indicated that, at the reaction condition of 320 °C, 5.5 h, and the In/Sn mole ratio of 90:10, the obtained nanoparticles had high monodispersity and crystallinity, and the resulting NF had good stability and dispersion. Furthermore, when tuning single reaction condition, the morphology and Sn doping of nanoparticles tended to be a regular change. However, the change of single reaction condition was not enough to obtain the ITO NFs with good stability and dispersion. The ITO NFs obtained in present study had high thermal stability, and showed different high-temperature characteristics in the Ultraviolet and visible (UV–Vis) and near infrared (NIR) regions.