• Energy Research
• US close

The dynamic dispatch (DD) of battery energy storage systems (BESSs) in microgrids integrated with volatile energy resources is essentially a multiperiod stochastic optimization problem (MSOP). Because the life span of a BESS is significantly affected by its charging and discharging behaviors, its lifecycle degradation costs should be incorporated into the DD model of BESSs, which makes it non-convex. In general, this MSOP is intractable. To solve this problem, we propose a reinforcement learning (RL) solution augmented with Monte-Carlo tree search (MCTS) and domain knowledge expressed as dispatching rules. In this solution, the Q-learning with function approximation is employed as the basic learning architecture that allows multistep bootstrapping and continuous policy learning. To improve the computation efficiency of randomized multistep simulations, we employed the MCTS to estimate the expected maximum action values. Moreover, we embedded a few dispatching rules in RL as probabilistic logics to reduce infeasible action explorations, which can improve the quality of the data-driven solution. Numerical test results show the proposed algorithm outperforms other baseline RL algorithms in all cases tested.

The dynamic dispatch (DD) of battery energy storage systems (BESSs) in microgrids integrated with volatile energy resources is essentially a multiperiod stochastic optimization problem (MSOP). Because the life span of a BESS is significantly affected by its charging and discharging behaviors, its lifecycle degradation costs should be incorporated into the DD model of BESSs, which makes it non-convex. In general, this MSOP is intractable. To solve this problem, we propose a reinforcement learning (RL) solution augmented with Monte-Carlo tree search (MCTS) and domain knowledge expressed as dispatching rules. In this solution, the Q-learning with function approximation is employed as the basic learning architecture that allows multistep bootstrapping and continuous policy learning. To improve the computation efficiency of randomized multistep simulations, we employed the MCTS to estimate the expected maximum action values. Moreover, we embedded a few dispatching rules in RL as probabilistic logics to reduce infeasible action explorations, which can improve the quality of the data-driven solution. Numerical test results show the proposed algorithm outperforms other baseline RL algorithms in all cases tested.

The integration of volatile distributed energy resources (DERs) brings new challenges for the active distribution network maintenance scheduling (DN-MS). Conventionally, the DN-MS is formulated as a deterministic optimization model without considering the uncertainties of DERs. In this paper, the DN-MS is formulated as a multistage stochastic optimization problem, which is cast as a stochastic mixed-integer nonlinear programming model. It aims to reduce the total maintenance cost constrained by the reliability indices. To capture the operational characteristics of active distribution networks, the uncertainties of DERs and post-outage operation strategies of switching devices are incorporated into the model. In general, this type of model is intractable and mainly solved by heuristic search methods with low efficiency. Recently, Monte-Carlo tree search (MCTS) is emerging as a scalable and promising reinforcement learning approach. We propose a stochastic MCTS solution to this problem. In the tree search procedure, a sample average approximation technique is developed to estimate multistage maintenance costs considering uncertainties. To speed up the MCTS, the complicated constraints of the original problem are transformed to penalty or heuristics functions. This approach can asymptotically approximate the optimum with promising computation efficiency. Numerical test results demonstrate the superiority of the proposed method over benchmark methods.

The integration of volatile distributed energy resources (DERs) brings new challenges for the active distribution network maintenance scheduling (DN-MS). Conventionally, the DN-MS is formulated as a deterministic optimization model without considering the uncertainties of DERs. In this paper, the DN-MS is formulated as a multistage stochastic optimization problem, which is cast as a stochastic mixed-integer nonlinear programming model. It aims to reduce the total maintenance cost constrained by the reliability indices. To capture the operational characteristics of active distribution networks, the uncertainties of DERs and post-outage operation strategies of switching devices are incorporated into the model. In general, this type of model is intractable and mainly solved by heuristic search methods with low efficiency. Recently, Monte-Carlo tree search (MCTS) is emerging as a scalable and promising reinforcement learning approach. We propose a stochastic MCTS solution to this problem. In the tree search procedure, a sample average approximation technique is developed to estimate multistage maintenance costs considering uncertainties. To speed up the MCTS, the complicated constraints of the original problem are transformed to penalty or heuristics functions. This approach can asymptotically approximate the optimum with promising computation efficiency. Numerical test results demonstrate the superiority of the proposed method over benchmark methods.

The coordinated solution of AC optimal power flow (ACOPF) in the integrated transmission and distribution grid utilizes controllable energy resources in distribution grids for introducing additional operational economy and security benefits of power grids. However, it is rather impractical to compute the ACOPF centrally considering the information privacy in the operation of transmission and distribution grids. Due to the large capacity of distributed generators integrated to distribution grids, existing decentralized methods may encounter numerical problems and fail to converge when nonlinear ACOPF models are applied. In this paper, we propose a decentralized method to solve the ACOPF for integrated transmission and distribution grids. The proposed ACOPF model is characterized through polar-coordinate equations and branch flow equations for transmission grids and distribution grids respectively. A distribution-cost-correction framework is developed for a fast-convergent solution which is based on approximated distribution cost functions. A rigorous proof is provided for convergence and numerical simulations are analyzed for standard IEEE systems which demonstrate the advantages of the coordinated approach in reducing generation dispatch costs and mitigating overvoltage problems. The paper validates through simulations that the accuracy, computational efficiency, and scalability of the proposed approach are superior to those of traditional methods.

The coordinated solution of AC optimal power flow (ACOPF) in the integrated transmission and distribution grid utilizes controllable energy resources in distribution grids for introducing additional operational economy and security benefits of power grids. However, it is rather impractical to compute the ACOPF centrally considering the information privacy in the operation of transmission and distribution grids. Due to the large capacity of distributed generators integrated to distribution grids, existing decentralized methods may encounter numerical problems and fail to converge when nonlinear ACOPF models are applied. In this paper, we propose a decentralized method to solve the ACOPF for integrated transmission and distribution grids. The proposed ACOPF model is characterized through polar-coordinate equations and branch flow equations for transmission grids and distribution grids respectively. A distribution-cost-correction framework is developed for a fast-convergent solution which is based on approximated distribution cost functions. A rigorous proof is provided for convergence and numerical simulations are analyzed for standard IEEE systems which demonstrate the advantages of the coordinated approach in reducing generation dispatch costs and mitigating overvoltage problems. The paper validates through simulations that the accuracy, computational efficiency, and scalability of the proposed approach are superior to those of traditional methods.