Abstract The occurrence of non-equilibrium process of steam condensation has a significant effect on the efficiency of low pressure part steam turbine. To investigate the phenomenon of non-equilibrium condensation of supersonic nozzle including several self-excited oscillating modes, a full Navier-Stokes viscous laminar model for non-equilibrium condensing steam flow was established and validated by experiments and theory. The flow characteristics of pressure oscillation and velocity phase diagrams of different self-excited oscillating modes were analyzed. Finally, the distinct distributions of mass fluxes for both core flow field and viscous boundary layer of the condensing steam flow were discussed further. The results showed the relative variation of the displacement thickness of throat boundary layer is up to 55.73% which is significant.