Abstract There is pressing need to explore the fundamental link between the mixing efficiency and the entrainment performance for ensuring an efficient working of ejector in the associated energy systems. This study details where, how and to what extent the primary and entrained flows mix inside the steam ejector through a novel ejector model with species transport. The evolution laws of the mixing layer growth, the entrainment performance and the mass transfer of two streams are systematically discussed under various operational conditions. Further, their fundament links are clearly presented in a form of functional relation. The key findings emerged from this study: The mixing begins over a fluctuating mixing layer since the mixing chamber entrance. In the situation of critical operational models, the mixing layer experiences a fluctuating and exponential growth successively, but its development is severely obstructed under the subcritical models. Additionally, the entrainment ratio and the non-mixing length ratio are highly linear correlation with the mass transfer ratio. The variations of operational parameters essentially change the relative mass transfer capacity of the entrained flow. The more easily the entrained flow mixes into the primary jet flow, a faster mixing layer growth and higher entrainment performance are to be gained.