Sichuan, a hilly area in southwestern China, is recommended as a bioethanol production base because of its abundant crop residue resources. However, removing the crop straw for bioethanol may negatively affect soil fertility and productivity due to the local purple soil vulnerability. To explore the impact of crop residue removal on soil fertility and productivity and meet the needs of sustainable agriculture, we conducted a crop residue removal experiment by measuring the soil organic carbon (SOC), total nitrogen (TN), and total phosphorus (TP) contents, and crop yield in the purple soil region in southwest China. Soil erosion was also simulated by Revised Universal Soil Loss Equation version 2 (RUSLE 2). The results showed that soil erosion increased with the increase of the straw removal rate. Compared with 0% removal treatment, the SOC content reduced at other removal rate treatments, especially for long-term residue removal. The effect of residue removal on soil TN and TP was not consistent within one year. After two years, residue removal greater than 25% caused a decrease in TN by 1.6–3.7%, and straw removal greater than 50% caused a TP decrease by 8.5–9.3%. More than 25% of the residue removed reduced maize and canola yields, and TN and TP content. However, all crop residue removal treatments resulted in SOC content reduction and soil erosion deterioration. In conclusion, crop residue removal was not recommended due to agricultural sustainability in Sichuan, China.