Africa’s lands are largely vulnerable and threatened by soil degradation and low water availability, especially in semi-arid and arid regions, limiting crop and livestock productivity and farmer livelihood options. Therefore, in African agricultural lands, adopting/improving measures that conserve soil and water resources is crucial. This review aims to provide an update on soil and water conservation (SWC) in terms of farmer practices and research actions and explore how SWC technologies and practices represent a pathway to build or re-establish soil health and enhance sustainable agriculture in Africa. It also aims to increase knowledge on best-fit SWC approaches. Soil conservation, which includes measures of controlling soil erosion and maintaining or improving soil fertility, is inseparable from water conservation. On agricultural lands, the two are typically co-addressed. Increasing plant biomass production through improved water, crop and soil management practices, and managing this biomass judiciously, have direct and indirect impacts on conserving soils and water resources, particularly in drylands. This study focuses on rainfed agricultural systems. We discuss the barriers and challenges to scaling up best-bet SWC technological and management options. Moreover, we show that options, such as Conservation Agriculture (CA), Agroforestry (AF), as well as integrated soil fertility management (ISFM) and field-scale rainwater harvesting (RWH), remain promising for the preservation and improvement of soil health in Africa’s farmlands and improving the resilience of agrosystems to climate change and variability as well as droughts.