Sedentary/inactive lifestyle leads middle-aged and older adults to metabolic syndrome and frailty. Capsinoids from nonpungent chili pepper cultivar have been reported to reduce body fat mass, promote metabolism, and improve unidentified complaints of chills. Additionally, they have an anti-inflammation effect; therefore, we hypothesized that continuous oral ingestion of capsinoids alleviates age-related inflammation in the brain and improves the physical activity (PA) in middle-aged and older adults. In our double-blind human study, 69 participants (17 male, 52 female; mean age: 74.1 ± 7.7 years; range: 52–87 years) were administered either 9 mg of capsinoids which were extracted from pepper fruit variety CH-19 Sweet (Capsicum anuum L.) (CP group), or a placebo (PL group) daily over a 3 month period. In an animal study, PA and inflammation-related mRNA expression in the brain were examined in 5-week (young) and 53-week (old) aged mice fed a diet with or without 0.3% dihydrocapsiate, a type of capsinoids, for 12 weeks. In a human study, capsinoids intake did not increase the amount of light-to-moderate PA less than 6.0 metabolic equivalents (METs) (CP: 103.0 ± 28.2 at baseline to 108.2 ± 28.3 at 12 weeks; PL: 104.6 ± 19.8 at baseline to 115.2 ± 23.6 at 12 weeks, METs × hour/week); however, in participants exhibiting an inactive lifestyle, it showed significant increase (CP: 84.5 ± 17.2 at baseline to 99.2 ± 24.9 at 12 weeks; PL: 99.7 ± 23.3 at baseline to 103.8 ± 21.9 at 12 weeks). The energy expenditure in physical activity also improved in the inactive CP group (CP: 481.2 ± 96.3 at baseline to 562.5 ± 145.5 at 12 weeks; PL: 536.8 ± 112.2 at baseline to 598.6 ± 127.6 at 12 weeks; kcal/day). In all participants, CP showed reduced waist circumference, percent body fat, and visceral fat volume; in addition, chills were eased in subjects aged 80 years and older. The older mice fed capsinoids showed increased locomotion activity, decreased inflammation, and oxidative stress in the brain. The results suggest that the continuous oral ingestion of capsinoids gains PA through anti-inflammation effect in the brain as well as reduces fat accumulation and chills in inactive and older humans.