Current European agriculture policies and strategies are aimed to boost the transition from the predominant conventional to a more environmentally friendly agriculture model. As part of this, it is crucial to identify the barriers that exist to implementing crop-specific management practices so that appropriate mitigating steps to overcome these can be executed. Participatory action research, where farmers are research actors rather than objects, is essential to identify the main barriers farmers have to face. The objective of this study was to identify the main barriers to the adoption of a combination of sustainable management practices in olive cropping in Southern Spain. A 20-item questionnaire was designed and responded to by 200 Spanish olive farmers. Exploratory and confirmatory factor analyses were used to assess the factor structure. The final best fit model included 14 items that were grouped in the following four barriers or facilitators: “lack of training/formation”, and “lack of economic/policy support” as barriers; “wellbeing–nature connection” and “environmental impact awareness” as facilitators. The mean scores on the different factors were higher than the theoretical mean, so the identification of the two barriers and the two facilitators for implementing sustainable practices was robust. Farmers, especially women, are concerned about their knowledge limitation in implementing sustainable management practices, which opens a window of opportunity for specific actions (i.e., training and demonstrative events) to be taken for accomplishing the agriculture sector transformation.