DRIVER DROWSINESS DETECTION SYSTEM THROUGH FACIAL EXPRESSION USING CONVOLUTIONAL NEURAL NETWORKS (CNN)
DOI:
https://doi.org/10.24191/mjoc.v8i1.20286Keywords:
Convolutional Neural Network (CNN), Deep Learning (DL), Driver Drowsiness, Facial Expression, FatigueAbstract
Driver drowsiness or fatigue is a significant factor that causes road accidents each year and considerably affects road safety. According to the World Health Organization (WHO), drowsy driving may contribute to approximately 6% of fatal and severe road accidents. To overcome this problem, we present a state-of-the-art, real-time drowsiness detection system, which exploits innovative deep-learning techniques to evaluate facial expressions. Our system analyzes not just the driver's eyes, mouth, and head rotation pose with front angles but also left and right yaw angles up to 90° to ensure the driver's safety. We gathered a dataset from public stock image websites, and manual image captures to develop the system. After processing the dataset, we extracted a wide range of features, which we fed into a deep convolutional neural network (CNN) algorithm. Specifically, we employed three different CNN algorithms which are EfficientDet D0, SSD MobileNet V2, and SSD ResNet50 V1, to classify the driver's drowsiness status using the facial key attributes in real time. Our results show that the SSD ResNet50 V1 model exhibited the highest accuracy and consistency in detecting driver drowsiness, underscoring the potential of our innovative system in promoting road safety. Our future work will focus on fine-tuning the approach to enhance its accuracy and performance.
References
Al-Azzoa, F., Mohammed, A., & Milanovab, M. (2018). Human Related-Health Actions Detection using Android Camera based on TensorFlow Object Detection API. International Journal of Advanced Computer Science and Applications, 9(10). https://doi.org/10.14569/ijacsa.2018.091002.
Can, D. B. G. E. S. (2010). Save your life this Memorial Day. National Sleep Foundation (NSF): Arlington, VA, USA.
Cireşan, D., Meier, U., Masci, J., & Schmidhuber, J. (2012). Multi-column deep neural network for traffic sign classification. Neural Networks, 32, 333–338. https://doi.org/10.1016/j.neunet.2012.02.023.
Deng, L. & Yu, D. (2014). Deep Learning: Methods and Applications. Foundations and Trends® in Signal Processing, 7(3-4), 197–387. https://doi.org/10.1561/2000000039
Fan, J., Ma, C., & Zhong, Y. (2020, April 19). A selective overview of deep learning. PubMed Central (PMC). Retrieved February 12, 2023, from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8300482/.
Fathabadi, F. R., Grantner, J. L., Abdel-Qader, I., & Shebrain, S. A. (2022). Box Trainer Assessment System with Real-Time Multi-Class Detection and Tracking of Laparoscopic Instruments, using CNN. Acta Polytechnica Hungarica, 19(2), 7–27. https://doi.org/10.12700/aph.19.2.2022.2.1.
Gad, A. (2021). Evaluating Object Detection Models Using Mean Average Precision - KDnuggets. KDnuggets. Retrieved February 9, 2023, from https://www.kdnuggets.com/evaluating-object-detection-models-using-mean-averageprecision.html.
Garcia-Venegas, M., Mercado-Ravell, D. A., & Carballo-Monsivais, C. A. (2021). On the safety of vulnerable road users by cyclist orientation detection using Deep Learning Machine Vision and Applications, 32(5), 109. https://doi.org/10.1007/s00138-021-01231-4.
Gulhane, M., & Mohod, P. S. (2014). Intelligent fatigue detection and automatic vehicle control system. arXiv. https://doi.org/10.48550/arXiv.1407.2412.
Halim, Z., & Shuhidan, S. M. (2022). Towards development of robust machine learning model for Malaysian corporation: A systematic review of essential aspects for corporation credit risk assessment. Malaysian Journal of Computing (MJoC), 7(1), 1011-1026.
Han, M., Wang, Q., Zhang, T., Wang, Y., Zhang, D., & Xu, B. (2023). Complex dynamic neurons improved spiking transformer network for efficient automatic speech recognition. arXiv. https://doi.org/10.48550/arXiv.2302.01194.
He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep Residual Learning for Image Recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770–778. https://doi.org/10.1109/cvpr.2016.90.
Hosang, J., Benenson, R., Dollar, P., & Schiele, B. (2016). What Makes for Effective Detection Proposals? IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(4), 814–830. https://doi.org/10.1109/tpami.2015.2465908.
Hu, X., & Huang, B. (2020). Face Detection based on SSD and CamShift. 2020 IEEE 9th Joint International Information Technology and Artificial Intelligence Conference (ITAIC). https://doi.org/10.1109/itaic49862.2020.9339094.
Jie, Z., Mahmoud, M., Stafford-Fraser, Q., Robinson, P., Dias, E. and Skrypchuk, L. (2018) Analysis of Yawning Behaviour in Spontaneous Expressions of Drowsy Drivers. 13th IEEE International Conference on Automatic Face & Gesture Recognition (pp. 571-576). IEEE.
Joshi, A., Kyal, S., Banerjee, S., and Mishra, T. (2020). In-the-wild Drowsiness Detection from Facial Expressions. 2020 IEEE Intelligent Vehicles Symposium (IV), 2020 IEEE Conference on (pp. 207-212). IEEE.
Kumar, R., Rathore, H., Agrawal, P., & Gupta, P. (2021). Drowsiness Detection Using ViolaJones Object Detection Algorithm for Real-Time Data. Advances in Intelligent Systems and Computing, 369–380. https://doi.org/10.1007/978-981-16-0171-2_35.
Leblond, R., Alayrac, J.-B., Osokin, A., & Lacoste-Julien, S. (2018). Searnn: Training runs with global-local losses. arXiv. https://doi.org/10.48550/arXiv.1706.04499.
Lin, T.-Y., Dollár, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017). Feature Pyramid Networks for Object Detection. ArXiv:1612.03144 [Cs]. https://arxiv.org/abs/1612.03144v2.
Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., & Berg, A. C. (2016). SSD: Single Shot MultiBox Detector. Computer Vision – ECCV 2016, 21–37. https://doi.org/10.1007/978-3-319-46448-0_2.
Longe, O. B., Mbarika, V., Kourouma, M., Wada, F., & Isabalija, R. (2010). Seeing beyond the surface, understanding and tracking fraudulent cyber activities. arXiv https://doi.org/10.48550/arXiv.1001.1993.
Lu, X., Kang, X., Nishide, S., & Ren, F. (2019). Object detection based on SSD ResNet. 2019 IEEE 6th International Conference on Cloud Computing and Intelligence Systems (CCIS). https://doi.org/10.1109/ccis48116.2019.9073753.
Maior, C.B., Moura, M.D., Santana, J.M., do, L.M., Nascimento, Macedo, J.B., Lins, I.D., & Droguett, E.L. (2018). Real-time SVM Classification for Drowsiness Detection Using Eye Aspect.
Mounika, D., Deepika, K. D., Varma, A.R.S.R., P., & Kesava, M. (2021). A driver drowsiness detection framework using deep learning heuristic. In Anil Neerukonda Institute of Technology and Sciences (UGC Autonomous). https://cse.anits.edu.in/projects/projects2021A5.pdf.
Reddi, S., & Eswar, G. V. (2021). Fake news in social media recognition using Modified Long Short-Term Memory network. Security in IoT Social Networks, 205-227. https://doi.org/10.1016/b978-0-12-821599-9.00009-1.
Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018). MobileNetV2: Inverted Residuals and Linear Bottlenecks. ArXiv.org. https://arxiv.org/abs/1801.04381.
Sarmadi, A., Fu, H., Krishnamurthy, P., Garg, S., & Khorrami, F. (2022). Privacy-preserving collaborative learning through feature extraction. arXiv. https://doi.org/10.48550/arXiv.2212.06322.
Tan, M., Pang, R., & Le, Q. V. (2020). EfficientDet: Scalable and Efficient Object Detection. ArXiv:1911.09070 [Cs, Eess]. https://arxiv.org/abs/1911.09070v7.
Taqi, A. M., Al-Azzo, F., Awad, A. E., & Milanova, M. (2019). Skin Lesion Detection by Android Camera based on SSD- MobileNet and TensorFlow Object Detection API. Www.semanticscholar.org. https://www.semanticscholar.org/paper/Skin-LesionDetection-by-Android-Camera-based-on-Taqi-Al-Azzo/776c0b3620d795bec5e940bff2d846bcc0c8f815.
Vesselenyi, T., Moca, S., Rus, A., Mitran, T., & Tătaru, B. (2017). Driver drowsiness detection using ANN image processing. IOP Conference Series: Materials Science and Engineering, 252, 012097. https://doi.org/10.1088/1757-899x/252/1/012097.
Vijayan, V. & Sherly, E. (2019). Real-time detection system of driver drowsiness based on representation learning using deep neural networks. Journal of Intelligent & Fuzzy Systems, 36(3), 1977–1985.
Yang, C., Wang, X., & Mao, S. (2020). Unsupervised Drowsy Driving Detection With RFID. IEEE Transactions on Vehicular Technology, 69(8), 8151–8163. https://doi.org/10.1109/tvt.2020.2995835.
Yu, H., Chen, C., Du, X., Li, Y., Rashwan, A., Hou, L, Jin, P., Yang, F., Liu, F., Kim, J., and Li, J. (2020). TensorFlow Model Garden. Available at: https://github.com/tensorflow/models.
Yu, J., Seidel, R., & Hirtz, G. (2019). OmniPD: One-Step Person Detection in Top-View Omnidirectional Indoor Scenes. Current Directions in Biomedical Engineering, 5(1), 239–244. https://doi.org/10.1515/cdbme-2019-0061.
Zocco, F., Huang, Ching-I., Wang, H.-C., Khyam, M. O., & Van, M. (2022). Towards More Efficient EfficientDets and Low-Light Real-Time Marine Debris Detection. ArXiv:2203.07155 [Cs]. https://arxiv.org/abs/2203.07155.
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Nipa Das Gupta, Rajesvary Rajoo, Patricia Jayshree Jacob
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
