The restrictions and social distancing measures due to the COVID-19 pandemic have created many obstacles in the practice of outdoor physical activity (OPA) throughout the world, particularly for the most vulnerable people, such as those with disabilities. The aim of this study was to explore the impact of pandemic-related limitations on the OPA of an Italian cohort of people with disabilities practicing sports. A retrospective observational study was conducted using an online survey. The questionnaire was distributed to 121 disabled athletes who practiced different outdoor physical activities. A total of 96 completed the survey, which collected demographic data, information about daily outdoor physical activity and sports habits, and about physical and psychological health before and during the pandemic. The frequency of daily OPA per week, along with the hours of physical activity, significantly decreased during the pandemic compared to those of the year before (p < 0.0001). A statistically significant deterioration was also found in the physical and mental well-being of disabled athletes during the pandemic (p < 0.0001) when compared to those from the year before the advent of COVID-19. This research demonstrated the negative impact of COVID-19 restrictions on OPA levels and on the physical and mental well-being of athletes with disabilities. It also highlighted a new challenge regarding the sustainability and integration of the national health system, demonstrating the necessity of improving the consistent accessibility of people with disabilities to OPA, both under normal conditions and emergency situations, in order to guarantee their psychophysical well-being.