Q-Learning with Eligibility Traces to Solve Non-Convex Economic Dispatch Problems

Authors: Mohammed I. Abouheaf, Sofie Haesaert, Wei-Jen Lee, Frank L. Lewis

Abstract:

Economic Dispatch is one of the most important power system management tools. It is used to allocate an amount of power generation to the generating units to meet the load demand. The Economic Dispatch problem is a large scale nonlinear constrained optimization problem. In general, heuristic optimization techniques are used to solve non-convex Economic Dispatch problem. In this paper, ideas from Reinforcement Learning are proposed to solve the non-convex Economic Dispatch problem. Q-Learning is a reinforcement learning techniques where each generating unit learn the optimal schedule of the generated power that minimizes the generation cost function. The eligibility traces are used to speed up the Q-Learning process. Q-Learning with eligibility traces is used to solve Economic Dispatch problems with valve point loading effect, multiple fuel options, and power transmission losses.

Keywords: Economic Dispatch, Non-Convex Cost Functions, Valve Point Loading Effect, Q-Learning, Eligibility Traces.

Digital Object Identifier (DOI): doi.org/10.5281/zenodo.1088772

References:

[1] A. J. Wood and B. F. Wollenberg, Power Generation, Operation, and Control. New York: Wiley, 1996.
[2] S. Duman, U. Guvenc, and N. Yorukeren, “Gravitational search algorithm for economic dispatch with valvepoint effects,” Int. Rev. Elect. Eng., vol. 5(6), pp.2890–2895, 2010.
[3] N. Amjady, and H. Nasiri-Rad, “Economic dispatch using an efficient real-coded genetic algorithm,” IET Gen. Trans. Dist., vol. 3(3), pp. 266-278, 2009.
[4] N. Amjady, and H. Nasiri-Rad, “Solution of nonconvex and non smooth economic dispatch by a new Adaptive Real Coded Genetic Algorithm,” Exp. Sys. with Appl., pp.5237-5239e45, 2010.
[5] D. Lukman, K. Walshe, and T. R. Blackburn, “Loss Minimisation in Industrial Power System Operation,” Australasian Universities Power Engineering Conference (AUPEC2000), Brisbane, pp. 15–20, 2000.
[6] W. M. Lin, and S. J. Chen, “Bid-based dynamic economic dispatch with an efficient interior point algorithm,” Elec. Pwr. and Enr. Sys., vol. (24), pp.51-57, 2002.
[7] J. Nanda, D. P. Kothari, and K. S. Lingamurthy, “Economic-emission load dispatch through goal programming techniques,” IEEE Trans. on Enr. Conv., vol. 3(1), pp. 26-32, 1998.
[8] G. P. Granelli, and M. Montagna, “Security-constrained economic dispatch using dual quadratic programming,” Elect. Pwr. Syst. Res., vol. (56), pp. 71-80, 2000.
[9] H. T. Yang, P. C. Yang, and C. L. Huang, “Evolutionary programming based economic dispatch for units with non-smooth fuel cost functions,” IEEE Trans. on Pwr. Sys., vol. 11(1), pp. 112-118, 1996.
[10] L. S. Coelho, and V. C. Mariani, “Combining of chaotic differential evolution and quadratic programming for economic dispatch optimization with valve-point effect,” IEEE Trans. Pwr. Sys., vol. 21(2), pp. 989–96, 2006.
[11] K. T. Chaturvedi, M. Pandit, and L. Srivastava, “Self-organizing hierarchical particle swarm optimization for nonconvex economic dispatch,” IEEE Trans. on Pwr. Sys., vol. 23(3), pp. 1079-87, 2008.
[12] K. P. Wong, and C. C. Fung, “Simulated Annealing based Economic Dispatch algorithm,” IEE Proc., C 140, vol. 6, pp. 509 – 515, 1993.
[13] W. M. Lin, F. S. Cheng, and M. T. Tsay, “An improved Tabu search for economic dispatch with multiple minima,” IEEE Trans. on Pwr. Sys., vol. 17(1), pp. 108-112, 2002.
[14] M. Vanitha and K. Thanushkodi, “An Efficient Technique for Solving the Economic Dispatch Problem using Biogeography Algorithm,” European J of Scientific Res., vol. 50(2), pp. 165-172.
[15] S. Sen and G. Weis, “Learning in multi-agent systems, in Multi-agent Systems: A Modern Approach to Distributed Artificial Intelligence,” Ed. Cambridge, MA: MIT Press, pp. 259–298, 1999.
[16] R. S. Sutton and A. G. Barto, Reinforcement Learning–An Introduction. Massachusetts: Cambridge, MIT Press, 1998.
[17] P. J. Werbos, Beyond Regression: New Tools for Prediction and Analysis in the Behavior Sciences. PhD Thesis, 1974.
[18] P. J. Werbos, “Approximate dynamic programming for real-time control and neural modeling. Handbook of Intelligent Control,” Ed. D.A. White and D.A. Sofge, New York: Van Nostrand Reinhold, 1992.
[19] I. Ahammed, E. A. Jasmin, F. R. Pazheri, and E. A. Al-Ammar, “Reinforcement Learning Solution to Economic Dispatch Using Pursuit Algorithm,” 6th IEEE-GCC Conference and Exhibition, Dubai, UAE, pp. 19-22, 2011.
[20] E. A. Jasmin, I. Ahamed, and V. P. Jagathyraj, “A Reinforcement Learning algorithm to Economic Dispatch considering transmission losses,” Proceedings of TENCON, 2008.
[21] I. Ciornei, and E. Kyriakides, “A GA – API solution for the economic dispatch of generation in power system operation,” IEEE Trans. on Pwr. Sys., pp. 1-9, 2011.
[22] C. L. Chinag, “Improved genetic algorithm for power economic dispatch of units with valve-point effects and multiple fuels,” IEEE Trans. Pwr. Sys., vol. 20(4), pp. 1690–1699, 2005.
[23] L. Busoniu, R. Babuska, and B. De Schutter, “Multi-agent reinforcement learning: A survey,” Proceedings of the 9th International Conference on Control, Automation, Robotics and Vision, Singapore; pp. 527-532, 2006.
[24] Z. L. Gaing, “Particle Swarm Optimization to Solving the Economic Dispatch Considering the Generator Constraints,” IEEE trans. on pwr. sys., vol. 18(3), pp. 1187–1195, 2003.
[25] U. Guvenc, S. Duman, B. Saracoglu, and A. Özturk, “A Hybrid GA-PSO Approach Based on Similarity for Various Types of Economic Dispatch Problems,” Kaunas University of Technology, Electronics And Electrical Engineering, vol. 2(108), pp. 109-114, 2011.
[26] A. Bhattacharya and P. K. Chattopadhyay, “Solving complex economic load dispatch problems using biogeography-based optimization,” Expert Sys. with Appl., vol. 37, pp. 3605-3615, 2010.
[27] S. R. Rayapudi, “An Intelligent Water Drop Algorithm for Solving Economic Load Dispatch Problem,” Int. J of Elect. and Elect. Eng, vol. 5(2), pp. 43-49, 2011.
[28] C. E. Lin and G. L. Viviani, “Hierarchical economic dispatch for piecewise quadratic cost functions,” IEEE Trans. Pwr. Appar. Syst., vol. 103(6), pp. 1170–1175, 1984.
[29] J. H. Park, Y. S. Kim, I. K. Eom, and K. Y. Lee, “Economic load dispatch for piecewise quadratic cost function using Hopfield neural network,” IEEE Trans. Pwr. Sys., vol. (8), pp. 1030–1038, 1993.
[30] K. Y. Lee, A. Sode-Yome, and J. H. Park, “Adaptive Hopfield neural network for economic load dispatch,” IEEE Trans. Pwr. Sys., vol. 13, pp. 519–526, 1998.
[31] T. Jayabarathi and G. Sadasivam, “Evolutionary programming based economic dispatch for units with multiple fuel options,” Eur. Trans. Electr. Pwr., vol. 10(3), pp. 167–170, 2000.
[32] P. S. Manoharan, P. S. Kannan, S. Baskar, and M. W. Ruthayarajan, “Penalty parameter-less constraint handling scheme based evolutionary algorithm solutions to economic dispatch,” IET Gener. Trans. Distrib., vol. 2(4), pp. 478–490, 2008.
[33] S. Khamsawang and S. Jiriwibhakorn, “DSPSO-TSA for economic dispatch problem with nonsmooth and noncontinuous cost functions,” Energy Conversion and Management, vol. 51(2), pp. 365-75, 2010.
[34] J. B. Park, Y. W. Jeong, J. R. Shin, K. Y. Lee KY, “An improved particle swarm optimization for nonconvex economic dispatch problems,” IEEE Transactions on Power Systems, vol. 25(1), pp. 156-166, 2010.
[35] A. Selvakumar and T. Khanushkodi, “Optimization using civilized swarm: solution to economic dispatch with multiple minima,” Elect. Pwr. Sys. Res., vol. 79(1), pp. 8-16, 2009.
[36] B. K. Panigrahi and V. R. Pandi, S. Das, “Adaptive particle swarm optimization approach for static and dynamic economic load dispatch,” Ener. Convr. and Mang., vol. 49(6), pp. 1407-15, 2008.