Abstract In order to satisfy the heightened emissions regulation and further enhance the performance of the engine, efforts on amelioration of the engine management system are of great significance besides using intelligent regression algorithms. Based on a hydrogen-enriched Wankel rotary engine, multiple engine operations with variable excess air ratios, variable ignition timing, and variable hydrogen enrichment have been carried out a series of engine calibration tests in detail. After recording the required experimental data, three different methods, including quadratic polynomial, artificial neural networks (ANN), and support vector machine (SVM) are applied to construct a multi-objective regression model which gives a unique insight into the mathematical relationship between the engine performance and the operation and control parameters. For the ANN, the effect of the number of nodes in the hidden layer on the regression performance was discussed, and the weight values of the ANN was optimized using a genetic algorithm. For the SVM, the effects of the kernel function and three optimization methods on regression performance were discussed. The results indicated that the SVM exhibited the best fitting results among the three methods. The optimal R2 for brake thermal efficiency, fuel energy flow rate, nitrogen oxide, carbon monoxide, and unburned hydrocarbon is 0.9877, 0.9840, 0.9949, 0.9937, and 0.9992, respectively. It is highly recommended that the SVM method as a generic methodology may be a new direction for nonlinear control system modeling of the Wankel engine.