Abstract In this study, a novel proton exchange membrane fuel cell (PEMFC) system is proposed, in which a composition adjustment organic Rankine cycle (CAORC) system is integrated. The waste heat from both PEMFC stack and after-burner is recovered by CAORC system to realize the maximum output power conversion from system waste heat. Based on 3D construction method of thermodynamic cycle, the system thermodynamic model is established to evaluate the performance of the proposed PEMFC-CAORC system. The effects of PEMFC operating temperature, current density and high evaporation temperature are analyzed. The system energy and exergy efficiencies are 39.44% and 47.60% at the PEMFC operating temperature of 358 K and current density of 0.6 A cm−2. As the PEMFC operating temperature rises, its performance can be further improved, and the system energy and exergy efficiencies at 368 K reach 40.36% and 48.72%. As the PEMFC current density increases, the system net power increases while both efficiencies drop. To improve the CAORC performance specifically, the evaporation temperature in the after-burner is suggested to be high enough, and genetic algorithm is used to optimize key parameters of the CAORC. The mixture of Benzene and Toluene are the optimum zeotropic fluid type for all optimization cases investigated, while its proportion, separation dryness and evaporation temperature vary in different PEMFC operating conditions. Compared with the original results, the CAORC thermal efficiency increases by over 4% with genetic algorithm optimization.