Modern power grids face the challenge of increasing renewable energy penetration that is stochastic in nature and calls for accurate demand predictions to provide the optimized power supply. Hence, increasing the self-consumption of renewable energy through demand response in households, local communities, and micro-grids is essential and calls for high demand prediction performance at lower levels of demand aggregations to achieve optimal performance. Although many of the recent studies have investigated both macro and micro scale short-term load forecasting (STLF), a comprehensive investigation on the effects of electrical demand aggregation size on STLF is minimal, especially with large sample sizes, where it is essential for optimal sizing of residential micro-grids, demand response markets, and virtual power plants. Hence, this study comprehensively investigates STLF of five aggregation levels (3, 10, 30, 100, and 479) based on a dataset of 479 residential dwellings in Osaka, Japan, with a sample size of (159, 47, 15, 4, and 1) per level, respectively, and investigates the underlying challenges in lower aggregation forecasting. Five deep learning (DL) methods are utilized for STLF and fine-tuned with extensive methodological sensitivity analysis and a variation of early stopping, where a detailed comparative analysis is developed. The test results reveal that a MAPE of (2.47–3.31%) close to country levels can be achieved on the highest aggregation, and below 10% can be sustained at 30 aggregated dwellings. Furthermore, the deep neural network (DNN) achieved the highest performance, followed by the Bi-directional Gated recurrent unit with fully connected layers (Bi-GRU-FCL), which had close to 15% faster training time and 40% fewer learnable parameters.