Abstract Industrial combustion systems such as power generation gas turbines, rocket motors, furnaces and boilers often face the problem of large-amplitude self-excited pressure oscillations that occur due to the onset of thermoacoustic instability. To prevent the onset of such instability, understanding the effects of fuel–air equivalence ratio ϕ, and fuel flow rate Vf on self-excited nonlinear thermoacoustic oscillations is of fundamental and practical importance. Experimental investigation of the roles of these parameters on triggering thermoacoustic instability in a swirl combustor has received very little attention. In this work, we design a swirling thermocoustic combustor and conduct a series of experimental tests. Autocorrelation and recurrence analysis of phase space trajectories reconstructed from the acoustic pressure time trace are performed. These experimental tests allow us to study the effect of the fuel–air equivalence ratio ϕ on the onset of thermoacoustic instability by varying the fuel volume flow rate Vf. We demonstrate that the fuel volume flow rate and the equivalence ratio play different but critical roles on generating thermoacoustic instability at different frequencies and amplitudes. Maximum sound pressure level can be as high as 135 dB. In addition, mode switching, (i.e. frequency swap) is found to occur between approximately ω3 ≈ 510 Hz and ω1 ≈ 170 Hz, depending on the equivalence ratio ϕ. Furthermore, the dominant frequency corresponding to the maximum amplitude is shown to be shifted by approximately 20%, as the fuel flow rate Vf is increased and the combustion condition is changed from lean to rich. These findings are quite useful for designing a feedback control strategy to stabilize an unstable combustor. The present work opens up an applicable means to design a stable swirling combustor.