Abstract Gasoline compression ignition combustion has been regarded as a potential technology for future vehicle power source. Understanding the relationship between gasoline properties and its auto-ignition behaviors is critical to the development of this new combustion technology. Aiming at this goal, a comprehensive parametric study was conducted to investigate the influence mechanism of gasoline octane number, fuel sensitivity and equivalence ratio on the combustion behaviors of an advanced compression engine. A hybrid analysis method was proposed in the study; a reaction kinetics model coupled with a new-developed gasoline surrogate mechanism was used to trace the engine’s combustion characteristics, while a density-based global sensitivity analysis method was applied to decode the composite effect of each factor. Results demonstrate two distinct combustion modes in low temperature conditions and give separation functions in the input space of three factors (R-Squares are above 97%). The reaction kinetics analysis points out the shift of dominant reaction groups is the main reason for incomplete combustion phenomenon. Then the global sensitivity result shows octane number is the dominant factor controlling the rate of the combustion heat release under all conditions (the averaged sensitivity index exceeds 0.7); fuel sensitivity only makes a considerable effect on combustion performances under low temperature regions, with sensitivity indices of 0.2–0.3. Besides, the local sensitivity of each factor on combustion behaviors at specific points is also given to understand the influence mechanism resulted from the variations of fuel properties. This study provides a useful method for getting deeper insights into the combustion dynamics and kinetic reaction in gasoline compression ignition engines.