Indoor temperature is relevant with regard to mortality and heat-related self-perceived health problems. The aim of this study was to describe the association between indoor temperature and physical performance in older adults. Eighty-one older adults (84% women, mean age 80.9 years, standard deviation 6.53) were visited every four weeks from May to October 2015 and additionally during two heat waves in July and August 2015. Indoor temperature, habitual gait speed, chair-rise performance and balance were assessed. Baseline assessment of gait speed was used to create two subgroups (lower versus higher gait speed) based on frailty criteria. The strongest effect of increasing temperature on habitual gait speed was observed in the subgroup of adults with higher gait speed (−0.087 m/s per increase of 10 °C; 95% confidence interval (CI): −0.136; −0.038). The strongest effects on timed chair-rise and balance performance were observed in the subgroup of adults with lower gait speed (2.03 s per increase of 10 °C (95% CI: 0.79; 3.28) and −3.92 s per increase of 10 °C (95% CI: −7.31; −0.52), respectively). Comparing results of physical performance in absentia of a heat wave and during a heat wave, habitual gait speed was negatively affected by heat in the total group and subgroup of adults with higher gait speed, chair-rise performance was negatively affected in all groups and balance was not affected. The study provides arguments for exercise interventions in general for older adults, because a better physical fitness might alleviate impediments of physical capacity and might provide resources for adequate adaptation in older adults during heat stress.