Abstract The tip leakage vortex (TLV) cavitation is investigated by a commercial Reynolds averaged Navier-Stokes (RANS) solver. A referenced test on a NACA0009 hydrofoil is used to validate the numerical simulation. Considering the local rotation characteristics of the vortical flow, a rotation-curvature corrected Shear-Stress-Transport model (SST-CC model) is applied to simulate the time-averaged turbulent flow. Compared to the original SST model, the SST-CC model improves the prediction of the velocity in TLV on the measured sections in downstream, and the vorticity and pressure features along the TLV trajectory are analysed numerically. In order to increase the prediction accuracy for the TLV cavitation, the empirical condensation coefficient (Fc) in Zwart's cavitation model is calibrated based on the referenced experiment. By introducing a vortex identification parameter (f∗) related to the strain rate tensor and the vorticity tensor, a relationship between the Fc and f∗ is built, and the effects of the rotational motion of the vortex on the cavity are embodied in a modified Zwart's cavitation model. Compared to the conventional Zwart's cavitation model, the modified cavitation model significantly improves the prediction of the TLV cavitation and gets a better agreement with the referenced test on different conditions with various gap widths.