The present cross-sectional study aimed to investigate the relationship between different dimensions of physical activity (PA) (i.e., work, sport, leisure) and health-related quality of life (HRQoL) (i.e., overall, physical component, mental component) in an adult lifespan sample of 381 active individuals (age range: 18–88 years; 38.8% men), while controlling for important covariates in terms of sex, age, education, and health profile regarding medical history. HRQoL was assessed using the 12-item Short-Form Health Survey (SF-12). Usual (i.e., previous 12 months) PA was assessed during face-to-face interviews using the Baecke questionnaire. Hierarchical regression analyses showed in Step 1 that the three PA dimensions work, sport, and leisure (entered simultaneously) together predicted 8%, 10%, and 4% of the variance in SF-12 total score, SF-12 physical component, and SF-12 mental component, respectively. In the final model, adjusting for sex, age, education, and health profile regarding medical history, sport emerged as the only PA dimension predicting SF-12 total score and the SF-12 physical component. In conclusion, health-policy targets at the community level should include the promotion of lifelong engagement in PA, especially sport, to allow the sustainability of HRQoL across the lifespan of our society.