Abstract In electrical engineering, the heat transfer can be enhanced by changing the thermophysical properties of insulating oils. In this paper, a single-phase power transformer with a nominal power of 5 kVA is subjected to a temperature rise test with three different transformer liquids. The first test is carried out with a novel gas-to-liquid transformer oil applied as a cooling and insulating medium. The other tests are conducted with ferrofluids based on this oil and MnZn ferrite nanoparticles of a low and a high nanoparticle concentration. The ferrofluids are characterized by magnetization curves, magnetic susceptibility and temperature-dependent magnetization measurements. The nanoparticle size distribution is determined from dynamic light scattering and the magnetization data. From the temperature rise profiles of the transformer at various inner locations, it has been found that the low-concentrated ferrofluid significantly reduces the transformer temperature rise. The enhanced cooling performance is ascribed to the thermomagnetic and natural convection, and increased thermal conductivity. The application of the ferrofluid with the high nanoparticle concentration resulted in a remarkable increase of the transformer temperature rise. The deteriorative cooling effect is attributed to the hindered natural and thermomagnetic convection due to the high ferrofluid magnetization and strong magnetic interaction of the ferrofluid with the magnetic field near the transformer core.