Automated Driving Deep Neural Network Model Accuracy and Performance Assessment in a Simulated Environment

Authors: David Tena-Gago, Jose M. Alcaraz Calero, Qi Wang

Abstract:

The evolution and integration of automated vehicles have become more and more tangible in recent years. State-of-the-art technological advances in the field of camera-based Artificial Intelligence (AI) and computer vision greatly favor the performance and reliability of Advanced Driver Assistance System (ADAS), leading to a greater knowledge of vehicular operation and resembling the human behaviour. However, the exclusive use of this technology still seems insufficient to control the vehicular operation at 100%. To reveal the degree of accuracy of the current camera-based automated driving AI modules, this paper studies the structure and behavior of one of the main solutions in a controlled testing environment. The results obtained clearly outline the lack of reliability when using exclusively the AI model in the perception stage, thereby entailing using additional complementary sensors to improve its safety and performance.

Keywords: Accuracy assessment, AI-Driven Mobility, Artificial Intelligence, automated vehicles.

References:

[1] S. SAE, “J3016 standard: Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles,” 2021.
[2] R. Lanctot et al., “Accelerating the future: The economic impact of the emerging passenger economy,” Strategy analytics, vol. 5, 2017.
[3] A. Toschi et al., “Characterizing perception module performance and robustness in production-scale autonomous driving system,” in IFIP ICNPC, 2019, pp. 235–247.
[4] A. H. M. Rubaiyat et al., “Experimental resilience assessment of an open-source driving agent,” in 2018 IEEE PRDC, 2018, pp. 54–63.
[5] J. Norden et al., “Efficient black-box assessment of autonomous vehicle safety,” arXiv preprint arXiv:1912.03618, 2019.
[6] T. Sato et al., “Hold tight and never let go: Security of deep learning based automated lane centering under physical-world attack,” arXiv preprint arXiv:2009.06701, 2020.
[7] Z. Zhong et al., “Detecting safety problems of multi-sensor fusion in autonomous driving,” arXiv preprint arXiv:2109.06404, 2021.
[8] M. Tan and Q. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” in International Conference on Machine Learning, 2019, pp. 6105–6114.
[9] H. Cui, V. Radosavljevic, F.-C. Chou, T.-H. Lin, T. Nguyen, T.-K. Huang, J. Schneider, and N. Djuric, “Multimodal trajectory predictions for autonomous driving using deep convolutional networks,” in 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019, pp. 2090–2096.
[10] “Comma.ai - comma2k19,” https://github.com/commaai/comma2k19, accessed: 06/08/2021.