Abstract The purpose of this study was the improvement of cold flow and lubricity characteristics of biodiesel produced from the palm-sesame oil blend. Extreme learning machine (ELM) and response surface methodology (RSM) techniques were used to model the production process and the input variables (time, catalyst amount, methanol to oil ratio, and duty cycle) were optimized using cuckoo search algorithm. The mean absolute percentage error (MAPE), coefficient of determination (R2), mean square error (MSE), root mean square error (RMSE), and standard error of prediction (SEP) were calculated to evaluate the performance of RSM and ELM. The results showed that ELM model had better performance in prediction than RSM model. The optimum yield of P50S50 biodiesel obtained was 96.6138% under operating parameters of time (38.96 min), duty cycle (59.52%), methanol to oil ratio (60 V/V %) and catalyst amount (0.70 wt%). The cold flow characteristics of P50S50 biodiesel are significantly improved like cloud point (7.89 °C), pour point (3.80 °C), and cold filter plugging point (- 1.77 °C) with better oxidation stability 6.89 h. The average coefficient of friction P50S50 biodiesel was lower than palm biodiesel (B100) and B10 commercial diesel by 2.29% and 12.37% respectively.