Soil fertility in smallholder farming areas is known to vary strongly on multiple scales. This study measures the sensitivity of the recorded satellite signal to on-farm soil fertility treatments applied to five crop types, and quantifies this fertilization effect with respect to within-field variation, between-field variation and field position in the catena. Plant growth was assessed in 5–6 plots per field in 48 fields located in the Sudano-Sahelian agro-ecological zone of southeastern Mali. A unique series of Very High Resolution (VHR) satellite and Unmanned Aerial Vehicle (UAV) images were used to calculate the Normalized Difference Vegetation Index (NDVI). In this experiment, for half of the fields at least 50% of the NDVI variance within a field was due to fertilization. Moreover, the sensitivity of NDVI to fertilizer application was crop-dependent and varied through the season, with optima at the end of August for peanut and cotton and early October for sorghum and maize. The influence of fertilizer on NDVI was comparatively small at the landscape scale (up to 35% of total variation), relative to the influence of other components of variation such as field management and catena position. The NDVI response could only partially be benchmarked against a fertilization reference within the field. We conclude that comparisons of the spatial and temporal responses of NDVI, with respect to fertilization and crop management, requires a stratification of soil catena-related crop growth conditions at the landscape scale.