In the literature, a lot is discussed about how agroforestry can achieve the mitigation, adaptation and productivity goals of climate-smart agriculture (CSA). However, this may be relatively too broad to assess the trade-offs and synergies of how specific agroforestry technologies or practices achieve the three pillars of CSA. Here, we provide an overview of how improved fallows (an agroforestry technology consisting of planting mainly legume tree/shrub species in rotation with cultivated crops) may achieve the goals of climate-smart agriculture in Sub-Saharan Africa (SSA). Our review showed that improved fallow systems have real potential to contribute to food security and climate change mitigation and adaptation in SSA. Under proper management, improved fallows can increase maize yields to about 6 t ha−1, which is comparable to conventional maize yields under fertilization. This is attributed to improved soil fertility and nutrient use efficiency. Although data was generally limited, the growing literature showed that improved fallows increased soil carbon sequestration and reduced greenhouse emissions. Further, as a multiple output land use system, improved fallows may increase fodder availability during dry periods and provide substantial biomass for charcoal production. These livelihood options may become important financial safety nets during off seasons or in the event of crop failures. This notwithstanding, the adoption of improved fallows is mainly in Southern and Eastern Africa, where over 20,000 farmers are now using Sesbania sesban, Tephrosia vogelii, and Cajanus cajan in two-year fallows followed by maize rotations. Land tenure issues, lack of social capital, and improved germplasm and accessions of fallow species have been cited as constraints to scaling up. However, development of seed orchards, nursery development, and the willingness of policy makers to create a policy environment that addresses market failures and alleviates disincentives should improve adoption and future scaling up.