Coffee silver skin, an organic residue from coffee production, demonstrates low solid fuel characteristics such as low bulk density and heating value, necessitating enhancements for solid fuel applications. Torrefaction in a flue gas environment (5% O2, 15% CO2, and a balance of N2, v/v) is more energy-efficient than inert torrefaction, using recovered flue gas to improve fuel quality and process efficiency. Three input factors were assessed: temperature (200, 250, and 300 °C), residence time (30, 45, and 60 min), and gas media (N2 and flue gas). Four performance metrics were evaluated: energy yield, upgrading energy index, specific energy consumption, and energy-mass co-benefit. Temperature significantly influenced most outcomes, except for energy-mass co-benefit, which was medium-dependent. Optimal torrefaction conditions achieving maximum energy yield (71.48%) and energy-mass co-benefit (5.30%) were identified at 200 °C for 30 min with flue gas. The torrefied material’s properties include moisture content, volatile matter, fixed carbon, and ash content of 3.03%, 69.24%, 27.04%, and 1.01%, respectively. Furthermore, the hydrophobicity of pelletized coffee silver skin notably increased under flue gas conditions, evident by a contact angle greater than 100°, indicating that flue gas torrefaction is a feasible approach for producing high-grade solid fuel.