Basin-wide multi-reservoir operation using reinforcement learning

Abstract

The analysis of large-scale water resources systems is often complicated by the presence of multiple reservoirs and diversions, the uncertainty of unregulated inflows and demands, and conflicting objectives. Reinforcement learning is presented herein as a new approach to solving the challenging problem of stochastic optimization of multi-reservoir systems. Conventional stochastic DP models have been applied to limited system representation requiring simplification and approximations that operators are unwilling to accept. The purpose of this study is to establish an optimization framework for realistic and reliable operation of multi-reservoir systems in order to reduce the gap between theoretical investigations and practical implementation. Reinforcement learning is a simulation-based technique rooted in dynamic programming. One of the reinforcement learning approaches called Q-Leaming avoiding requiring prior knowledge of the state transition probabilities in the system by direct use of historical data. The optimal control policy is learned based entirely on feedback mechanisms. In addition, reinforcement learning does not require synthetic streamflow generation and method for inferring rules required in implicit stochastic optimization approaches. The Keum River basin in Korea was chosen as a case study to demonstrate the applicability of reinforcement learning for basin wide reservoir operation. The Keum River basin consists of 12 sub basins and two major reservoirs, and the operation of this river basin includes water supply, flood control, hydropower generation, and instream flow requirements. These multiple objectives are combined into a single objective function for the dynamic programming optimization using the weighting method, assuming a unique performance measure for commensurating multi-objectives exists. A detailed simulation procedure for the Keum River basin is developed to accurately reflect the basin characteristics and consider all important component in the basin. The Q-Learning method is used for generating integrated monthly operation rules for the Keum River basin. The Q-Learning model is evaluated by comparing with implicit stochastic dynamic programming and sampling stochastic dynamic programming approaches. Evaluation of the stochastic basin-wide operational models considered several options relating to the choice of hydrologic state and discount factors as well as various stochastic dynamic programming models. The performance of Q-Learning model outperforms the other models in handling of uncertainty of inflows.