Abstract Nanofluid performed a very crucial role in nuclear power plant for improving heat transfer. Due to high thermal conductivity of nanofluid heat transfer capacity in nuclear reactor can be thousand times larger than conventional fluids like water. So the efficiency of nuclear reactor is improving and also reducing the thermal hydraulics problems. It opens a new gateway for gaining higher energy optimization. For improving the heat transfer in light water reactor, SiO2-water based nanofluids properties were used for numerical investigation. Heat transfer of nanofluid can be functions of volume concentration, physical properties and size of nanoparticles etc. Density, thermal conductivity, specific heat and viscosity were investigated and operated as a basic data for ANSYS 14. In this numerical investigation three value of weight concentration of nanoparticles in the range of 1%, 2% and 3% were used. A uniform heat flux of 1000 (kW/m2) is applied at the wall of annular rod. The numerical method which is available in CFD package of Ansys CFX 14 has been used here. Turbulence models of k–ω has been used in this numerical analysis. Validation of result has been done by analytical equations. Numerical investigation of various effect on characteristics of heat transfer in light water nuclear reactor by using SiO2-water based nanofluid is the key objective of the study. Coefficient of heat transfer and temperature of clad wall profiles are plotted without nanofluids and with nanofluids.