Social-ecological resilience (SER), understood as the capacity to prevent, react to, and mitigate crises that affect social-ecological systems, provides an integrative framework to analyze agricultural challenges. Based on this approach, key points that affect the sustainability of productive landscapes are addressed and evaluated, providing a baseline from which to improve farming systems at different scales. Hence, the aim of this work is to assess SER in potato crops in the Nariño area in southwestern Colombia, a region where strategies to increase resilience must be implemented. Following the methodology proposed by the UNU-IAS (2014), potato producers’ thoughts and perceptions were evaluated by implementing eleven workshops in seven municipalities. Five main integrative factors (twenty indicators of resilience) were examined and scored during the assessment: (1) governance and social equity, (2) livelihood and well-being, (3) knowledge and innovation, (4) landscape diversity and ecosystem protection, and (5) agrobiodiversity and sustainable natural resource management. Participants evaluated each indicator from 1 to 5 (1 being low performance and 5 extremely good performance). The results were calculated and averaged. Prior to the assessment, participatory techniques to generate collective reflection on resilience and landscape management were performed. The results showed that farmers rated SER resilience from low to moderate (from 2.5 to 3.2), with “well-being” (2.5) and “knowledge and innovation” (2.7) being the worst-rated factors. The data evidence deficiencies in all the indicators examined. Issues that constrain SER are related to the lack of capacity to create bio-industries, small livelihood portfolios, pollution, loss of natural areas (which impacts biodiversity and ecosystem services), and the loss of ancestral knowledge. The producers requested, as short-term actions, increases in technical assistance (to promote innovation and business initiatives) and farm diversification programs (to take advantage of their native potatoes diversity). They also agreed on the need for associative figures to enhance capacity-building among producers. These findings confirm deficiencies that minimize the sustainability of this system. Actions that impact positively almost all indicators are required to improve not only productivity but also the population’s well-being.