Demand response (DR) is an important technique to explore the demand-side flexibility. The wide deployment of smart meters makes it possible to quantify the baseline load. As an intermediate agent, demand response aggregator needs to obtain the aggregated baseline load (ABL) for the DR event. Previous studies about the household level estimation focus on the estimation method. However, for ABL estimation, customer division is an important issue. A major limitation is the mismatch between the objectives of segmentation and estimation. Therefore, this paper proposes a new closed-loop method for estimating the ABL, which utilizes the contextual bandit with policy gradient to link the segmentation with the estimation. As such, the ABL estimation accuracy can guide the segmentation to divide the customers. The segmentation and estimation optimize collaboratively to improve the ABL estimation accuracy. An ensemble method for combining network’s weights during the training process is proposed. Moreover, a pre-and post-event adjustment method is developed to further improve the estimation accuracy. Comprehensive comparisons demonstrate the proposed method can achieve the best estimation performance with regard to the MAPE and RMSE. It improves the estimation accuracy by 7% in terms of MAPE, and 11% in terms of RMSE.