Phenological events, such as the initiation and the end of seasonal growth, are thought to be under strong evolutionary control because of their influence on tree fitness. Although numerous studies highlighted genetic differentiation in phenology among populations from contrasting climates, it remains unclear whether local adaptation could restrict phenological plasticity in response to current warming. Seedling populations of seven deciduous tree species from high and low elevations in the Swiss Alps were investigated in eight common gardens located along two elevational gradients from 400 to 1,700 m. We addressed the following questions: are there genetic differentiations in phenology between populations from low and high elevations, and are populations from the upper elevational limit of a species' distribution able to respond to increasing temperature to the same extent as low-elevation populations? Genetic variation of leaf unfolding date between seedlings from low and high populations was detected in six out of seven tree species. Except for beech, populations from high elevations tended to flush later than populations from low elevations, emphasizing that phenology is likely to be under evolutionary pressure. Furthermore, seedlings from high elevation exhibited lower phenological plasticity to temperature than low-elevation provenances. This difference in phenological plasticity may reflect the opposing selective forces involved (i.e. a trade-off between maximizing growing season length and avoiding frost damages). Nevertheless, environmental effects were much stronger than genetic effects, suggesting a high phenological plasticity to enable tree populations to track ongoing climate change, which includes the risk of tracking unusually warm springs followed by frost.