Abstract The present work is aimed at optimizing the geometry of the curved trapezoidal winglets to enhance the overall performance of the heat exchanger. Prior to optimization, the most critical geometric entities are identified to reduce computational overheads. Based on the performance, three design parameters viz. arc radius (R), the angle subtended (θ), and winglet’s larger end height (h2) are chosen for optimization. By making use of the Latin hypercube sampling plan, the design of experiments has been conducted for the above mentioned variables. The required responses for different combinations of the independent variables – that include the Colburn factor (j) and friction factor (f) – are computed by making use of computational fluid dynamics. Both fluid and solid domains are discretized using hexahedral control volumes and the numerical simulations are performed by accounting the conjugate heat transfer approach. The SST k–ω (a 2-equation based) turbulence model is used as a closure model for Reynolds-averaged Navier-Stokes equations to evaluate the values of j and f. The data from DoE are used to train an artificial neural network for multiobjective optimization. Finally, the optimized data sets are generated using genetic algorithms and very encouraging results have been obtained. These Pareto front points range from an energy-efficient to a high-performance heat exchanger design. Therefore, the designers can make the choice(s) of the winglet geometry over a wide range depending on the desired performance of the heat exchanger.