Abstract This paper investigates the forced convection of alumina-water nanofluids within helical tubes, maintaining a constant wall temperature and assuming thermal equilibrium between the nanoparticles and the base fluid. The nanofluid model incorporates the effects of alumina (Al2O3) nanoparticle volume fraction, diameter, and temperature on thermophysical properties. The governing equations are solved using the Forward-Time Central-Space Finite Volume method in conjunction with the simple algorithm. Numerical results are validated against experimental data, demonstrating high accuracy. The study explores the effects of pitch size, curvature ratio, nanoparticle volume fraction, nanoparticle diameter, and Reynolds number on velocity contours, temperature profiles, secondary flow, thermophysical properties, friction coefficient, and heat transfer rate. Additionally, the figure of merit evaluates the impact of these parameters on the thermal performance of the system. The results indicate that an increase in Reynolds number and nanoparticle diameter negatively affects thermal performance, while higher nanoparticle volume fraction, curvature ratio, and pitch size enhance it. Furthermore, incorporating nanoparticles in straight tubes proves to be more advantageous compared to helical tubes. This study tested volumetric ratios of 1%, 2%, and 4%, which resulted in increases in heat transfer coefficients of 21%, 32%, and 43%, respectively, compared to pure water under similar conditions, such as Reynolds number and coil pitch.