To improve the design of the electricity infrastructure and the efficient deployment of distributed and renewable energy sources, a new paradigm for the energy supply chain is emerging, leading to the development of smart grids. There is a need to add intelligence at all levels in the grid, acting over various time horizons. Predicting the behavior of the energy system is crucial to mitigate potential uncertainties. An accurate energy prediction at the customer level will reflect directly in efficiency improvements in the whole system. However, prediction of building energy consumption is complex due to many influencing factors, such as climate, performance of thermal systems, and occupancy patterns. Therefore, current state-of-the-art methods are not able to confine the uncertainty at the building level due to the many fluctuations in influencing variables. As an evolution of artificial neural network (ANN)-based prediction methods, deep learning techniques are expected to increase the prediction accuracy by allowing higher levels of abstraction. In this paper, we investigate two newly developed stochastic models for time series prediction of energy consumption, namely Conditional Restricted Boltzmann Machine (CRBM) and Factored Conditional Restricted Boltzmann Machine (FCRBM). The assessment is made on a benchmark dataset consisting of almost four years of one minute resolution electric power consumption data collected from an individual residential customer. The results show that for the energy prediction problem solved here, FCRBM outperforms ANN, Support Vector Machine (SVM), Recurrent Neural Networks (RNN) and CRBM.