Abstract Once-Through Steam Generator (OTSG) is widely used in nuclear reactor system due to its advantages of compactness. This paper presents a double-tube Once-Through Steam Generator (OTSG) consisting of an outer straight tube and an inner helical tube. In the double-tube, the primary fluid is divided into two parts: one is in the inner tube and the other is in the shell side. This bi-channel flow brings difficulties in optimizing the flow distribution ratio of the primary water (FDRP), and affects the heat transfer and flow characteristics. In this paper, a nonlinear constrained optimization method was proposed to optimize the FDRP for keeping balance between maximizing the heat transfer rate and minimizing the pressure drop. A throttling device at the inlet of the lower pressure drop side is introduced to achieve the optimum FDRP. The theoretical analysis and analytical results show that our proposed method can be used in engineering practice for optimizing the primary flow.