Abstract Based on a retrofitted optical engine, high-speed photography technology is applied to obtain combustion flame patterns within the cylinder. The flame temperature and the space-time distribution of the characterize soot concentration (KL factor) are calculated using two-color method. Effects of coal-based synthetic fuel (CTL) and its mixture with oxygenated fuels on the combustion and soot generation process of the engine are experimentally studied. Two additional fuels are used in this study; one is alcohol-based fuel, the n-butanol and the other one is dimethyl carbonate (DMC). Result shows that when oxygenated fuels are mixed into the CTL, the ignition timing is postponed, and the ignition delay is prolonged. At the same time, with a shortened ignition duration caused by the additional oxygenated fuels, the flame area and flame brightness are significantly reduced. For the CTL and DMC blends, as the percent of DMC increases, both the combustion temperature within the cylinder and the KL factor are observed reducing in gradient, those are favorable for inhibiting formation of the soot. Comparison between the two mixed fuels shows that, with a same oxygen fraction, the temperature distribution of the CLT/n-butanol blends presents a bimodal form, and the average in-cylinder temperature is lower than that of the CTL/DMC blends, these characters are conductive to low-temperature combustion. With regards to the KL factor, the CTL/n-butanol mixture has a stronger effect on reducing the KL factor compared to that of the CTL/DMC blends.