Reinforcement Learning for Self Driving Racing Car Games

Authors: Adam Beaunoyer, Cory Beaunoyer, Mohammed Elmorsy, Hanan Saleh

Abstract:

This research aims to create a reinforcement learning agent capable of racing in challenging simulated environments with a low collision count. We present a reinforcement learning agent that can navigate challenging tracks using both a Deep Q-Network (DQN) and a Soft Actor-Critic (SAC) method. A challenging track includes curves, jumps, and varying road widths throughout. Using open-source code on Github, the environment used in this research is based on the 1995 racing game WipeOut. The proposed reinforcement learning agent can navigate challenging tracks rapidly while maintaining low racing completion time and collision count. The results show that the SAC model outperforms the DQN model by a large margin. We also propose an alternative multiple-car model that can navigate the track without colliding with other vehicles on the track. The SAC model is the basis for the multiple-car model where it can complete the laps quicker than the single-car model but has a higher collision rate with the track wall.

Keywords: Reinforcement learning, soft actor-critic, Deep Q-Network, Self-driving cars, artificial intelligence, Gaming.

References:

[1] R. Sutton and A. Barto, Reinforcement Learning An Introduction Cambridge, MA: The MIT Press, 2018, E-Book, Available: http://incompleteideas.net/book/RLbook2020.pdf
[2] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, and M. Leach, ”Mastering the game of go with deep neural networks and tree search”, Nature, vol. 529, no. 7587, 2017, doi: 10.1038/nature16961
[3] P. Wurman, S. Barrett, K. Kawamoto, J. MacGlashan, K. Subramanian, T. J. Walsh, R. Capobianco, A. Devlic, F. Eckert, F. Fuchs, L. Gilpin, P. Khandelwal, V. Kompella, H. Lin, P. MacAlpine, D. Oller, T. Seno, C. Sherstan, M. D. Thomure, and H. Kitano, ”Outracing champion Gran Turismo drivers with deep reinforcement learning,” Nature, vol. 602, pp. 223-228, 2022, doi: 10.1038/s41586-021-04357-7.
[4] N. Wang, X. Wang, P. Palacharla, and T. Ikeuchi, ”Cooperative autonomous driving for traffic congestion avoidance through vehicle-to-vehicle communications,” IEEE Vehicular Networking Conference (VNC), pp. 327-330, 2017 doi: 10.1109/VNC.2017.8275620.
[5] Y. Sun, ”Performance of Reinforcement Learning on Traditional Video Games,” 3rd International Conference on Artificial Intelligence and Advanced Manufacture (AIAM), pp. 276-279, 2021 doi: 10.1109/AIAM54119.2021.00063.
[6] L. A. Garza-Coello, M. M. Zubler, A. N. Montiel and C. Vazquez-Hurtado, ”AWS DeepRacer: A Way to Understand and Apply the Reinforcement Learning Methods,” IEEE Global Engineering Education Conference (EDUCON), pp. 1-3, 2023, doi: 10.1109/EDUCON54358.2023.10125166.
[7] J. McCalip, M. Pradhan and K. Yang, ”Reinforcement Learning Approaches for Racing and Object Avoidance on AWS DeepRacer,” IEEE 47th Annual Computers, Software, and Applications Conference (COMPSAC), pp. 958-961, 2023, doi: 10.1109/COMPSAC57700.2023.00129.
[8] T. Kohsuke, S. Yuta, O. Yuichi and S. Taro, ”Design of Reward Functions for RL-based High-Speed Autonomous Driving,” IEEE 15th International Symposium on Embedded Multicore/Many-core Systems-on-Chip (MCSoC) pp. 32-37, 2022, doi: 10.1109/MCSoC57363.2022.00015.
[9] F. Fuchs, Y. Song, E. Kaufmann, D. Scaramuzza, and P. Durr, ”Super-Human Performance in Gran Turismo Sport Using Deep Reinforcement Learning.” IEEE Robotics and Automation Letters, vol. 6, no. 3, pp. 4257–4264, 2021, doi: 10.1109/LRA.2021.3064284
[10] A. Kanervisto, C. Scheller and V. Hautam¨aki, ”Action Space Shaping in Deep Reinforcement Learning.” IEEE Conference on Games (CoG), Osaka, Japan, pp. 479-486, 2020, doi: 10.1109/CoG47356.2020.9231687.
[11] Y. Arjoune and S. Faruque, ”Double Deep Q-Learning and SAC Based Hybrid Beamforming for 5G and Beyond Millimeter-Wave Systems.” IEEE International Conference on Electro Information Technology (EIT), pp. 422-428, 2021, doi: 10.1109/EIT51626.2021.9491918.
[12] H. Goh, Y. Huang, C. Lim, D. Zhang, H. Liu, W. Dai, T. Kurniawan, and S. Rahman, ”An Assessment of Multistage Reward Function Design for Deep Reinforcement Learning-Based Microgrid Energy Management.” IEEE Transactions on Smart Grid, vol. 13, no. 6, pp 4300–4311, 2022, doi:10.1109/TSG.2022.3179567
[13] C. Lazaridis, I. Michailidis, G. Karatzinis, P. Michailidis, and E. Kosmatopoulos, ”Evaluating Reinforcement Learning Algorithms in Residential Energy Saving and Comfort Management,” Energies, vol. 1, no. 2, pp. 581-614, 2024, https://doi.org/10.3390/en17030581
[14] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing Atari with Deep Reinforcement Learning,” arXiv preprint arXiv:1312.5602, 2013 doi: 10.48550/arXiv.1312.5602
[15] Y. Dong, “Research on deep learning video game simulation algorithm based on Western Wind Music,” Entertainment Computing, vol. 46, p. 100566, May 2023. doi:10.1016/j.entcom.2023.100566
[16] N. A. Barriga, M. Stanescu, F. Besoain, and M. Buro, “Improving RTS game AI by supervised policy learning, tactical search, and deep reinforcement learning,” IEEE Computational Intelligence Magazine, vol. 14, no. 3, pp. 8–18, Aug. 2019. doi:10.1109/mci.2019.2919363
[17] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, ”Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.” In International conference on machine learning, pp. 1861-1870, 2018, doi: 10.48550/arXiv.1801.01290
[18] O. Aydogmus and M. Yilmaz, “Comparative analysis of reinforcement learning algorithms for Bipedal Robot locomotion,” IEEE Access, vol. 12, pp. 7490–7499, 2024. doi:10.1109/access.2023.3344393
[19] ”Wipeout.” Psygnosis, published by Sony Computer Entertainment, 1995.
[20] D. Szablewski, ”wipeout-rewrite.” Github repository, https://github.com/phoboslab/wipeout-rewrite
[21] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, and J. Schulman, ”OpenAI Gym.” OpenAI, 2016. Online. Available: https://github.com/openai/gym.
[22] A. Raffin, A. Hill, A. Gleeve, A.Kanervisto, and M. Ernestu, ”Stable Baselines3 : Reliable Reinforcement Learning Implementations.” Journal of Machine Learning Research, vol. 22, no. 268. 1-8, 2021. http://jmlr.org/papers/v22/20-1364.html
[23] S. Sivashangaran, A. Khairnar, and A. Eskandarian, “AutoVRL: A high fidelity autonomous ground vehicle simulator for SIM-to-real deep reinforcement learning,” IFAC-PapersOnLine, vol. 56, no. 3, pp. 475–480, 2023. doi:10.1016/j.ifacol.2023.12.069