Abstract Flow and heat transfer characteristics under rolling motion are extremely important to thermohydraulic analysis of offshore nuclear reactors. An experimental study was conducted in a heated rectangular channel to investigate flow and heat transfer in laminar–turbulent transitional flow regime under rolling motion. The results showed that the average friction factor and Nusselt number are higher than that of the corresponding steady flow as the flow rate fluctuates in transitional flow regime. Larger relative flow rate fluctuation was observed under larger rolling amplitude or higher rolling frequency. In the same manner, larger increases of average friction factor and Nusselt number were achieved under larger rolling amplitude or higher rolling frequency. The increases were mainly caused by the flow rate fluctuation through periodic breakdown of laminar flow and development of turbulence in laminar–turbulent transitional flow regime. First, turbulence, which enhances the rate of momentum and energy exchange, occurs near the crest of flow rate wave even the flow is still in laminar flow regime according to the average Reynolds number. Second, as a result of rapid increases of the friction and heat transfer with Reynolds number in transitional flow regime, the increases of the friction and the heat transfer near the crest of flow rate wave are larger than the decreases of them near the trough of flow rate wave, which also contributes to increases of average friction and heat transfer. Additionally, the effect of critical Reynolds number shift under unsteady flow and heating condition on flow and heat transfer was discussed.