Aggregations of demand response resources can provide a variety of services to the power grid. To utilize them effectively, for both planning problems and real-time control, we must estimate their flexibility. However, flexibility estimates are uncertain because of issues such as model error and forecasting error. In this paper, we present a model of uncertain flexibility and describe the many causes of uncertainty. We conduct two case studies, one for electric vehicle aggregations and one for air conditioner aggregations, in order to show specific examples that illustrate the causes and magnitude of uncertainty. We find that uncertainty can be very large for small load aggregations, when models do not capture enough of the underlying dynamics, or when forecasts of other quantities, such as ambient conditions, are bad. Although the focus of the paper is on understanding uncertainty, we also briefly discuss how one might use knowledge of uncertainty distributions in planning problems to derive closer-to-optimal results.