In-Air Hand Gesture Signature Recognition Using Multi-Scale Convolutional Neural Networks

Alvin Chuen - Multimedia University, 75450, Melaka, Malaysia
Khoh Wee How - Multimedia University, 75450, Melaka, Malaysia
Pang Ying Han - Multimedia University, 75450, Melaka, Malaysia
Yap Hui Yen - Multimedia University, 75450, Melaka, Malaysia


Citation Format:



DOI: http://dx.doi.org/10.30630/joiv.7.3-2.2359

Abstract


The hand signature is a unique handwritten name or symbol that serves as a proof of identity. Due to its practicality and widespread use, hand signature is still used by financial institutions as a means of verifying and validating the identity of their customers. The emergence of the COVID-19 global pandemic has raised hygiene concerns regarding the conventional touch-based hand signature recognition system, which often requires sharing the acquisition devices among the public. This paper presents in-air hand gesture signature recognition using convolutional neural networks to address this concern. We designed a shallow multi-scale convolutional neural network using 3x3 and 5x5 kernel filter sizes to extract features on different scales. The feature maps from these two filters are then concatenated to provide more robust features, which improve the model’s performance. The experiment results show that the proposed architecture outperforms other architectures, which obtained the highest accuracy of 93.00%. On the other hand, our architecture consumed significantly fewer computational resources, requiring only an average of 3 minutes and 33 seconds to train. Additionally, the performance of the proposed architecture could be further enhanced by integrating it with recurrent neural networks (RNN). This integrated architecture of convolutional recurrent neural networks (C-RNN) can capture spatio-temporal features simultaneously.


Keywords


Hand gesture signature; gesture recognition; inair signatures; convolutional neural networks.

Full Text:

PDF

References


A. Kholmatov and B. Yanikoglu, “Identity authentication using improved online signature verification method,†Pattern Recognit Lett, vol. 26, no. 15, pp. 2400–2408, 2005, doi: 10.1016/j.patrec.2005.04.017.

A. McCabe, J. Trevathan, and W. Read, “Neural Network-based Handwritten Signature Verification,†J Comput (Taipei), vol. 3, no. 8, 2008, doi: 10.4304/jcp.3.8.9-22.

R. Vera-Rodriguez, J. Fierrez, A. Morales, and J. Ortega-Garcia, “Dynamic signature recognition for automatic student authentication,†IATED, 2015.

S. Carlaw, “Impact on biometrics of Covid-19,†Biometric Technology Today, vol. 2020, no. 4, pp. 8–9, 2020, doi: 10.1016/S0969-4765(20)30050-3.

A. Buriro, R. Van Acker, B. Crispo, and A. Mahboob, “AirSign: A Gesture-Based Smartwatch User Authentication,†Proceedings - International Carnahan Conference on Security Technology, vol. 2018-Octob, pp. 20–21, 2018, doi: 10.1109/CCST.2018.8585571.

M. A. A. Haseeb and R. Parasuraman, “Wisture: RNN-based Learning of Wireless Signals for Gesture Recognition in Unmodified Smartphones,†pp. 1–10, 2017, [Online]. Available: http://arxiv.org/abs/1707.08569

L. G. Hafemann, R. Sabourin, and L. S. Oliveira, “Offline handwritten signature verification — Literature review,†2017 Seventh International Conference on Image Processing Theory, Tools and Applications (IPTA), vol. (pp. 1-8). 2017. doi: 10.1109/ipta.2017.8310112.

H.-C. Moon, S. Jang, K. Oh, and K.-A. Toh, “An In-Air Signature Verification System Using Wi-Fi Signals,†Proceedings of the 4th International Conference on Biomedical and Bioinformatics Engineering. pp. 133–138, 2017. doi: 10.1145/3168776.3168799.

G. Li, L. Zhang, and H. Sato, “In-air Signature Authentication Using Smartwatch Motion Sensors,†IEEE 45th Annual Computers, Software, and Applications Conference (COMPSAC). pp. 386–395, 2021. doi: 10.1109/COMPSAC51774.2021.00061.

R. Zhao, D. Wang, Q. Zhang, X. Jin, and K. Liu, “Smartphone-based Handwritten Signature Verification Using Acoustic Signals,†Proceedings of the ACM on Human-Computer Interaction, vol. 5, no. ISS. pp. 1–26, 2021. doi: 10.1145/3488544.

E. Guerra-Segura, A. Ortega-Pérez, and C. M. Travieso, “In-air Signature Verification System Using Leap Motion,†Expert Syst Appl, vol. 165, no. 113797, 2021, doi: 10.1016/j.eswa.2020.113797.

L. De Luisa, G. E. Hine, E. Maiorana, and P. Campisi, “In-Air 3D Dynamic Signature Recognition Using Haptic Devices,†International Workshop on Biometrics and Forensics (IWBF). IEEE, pp. 1–6, 2021. doi: 10.1109/iwbf50991.2021.9465089.

K. Kancharla, V. Kamble, and M. Kapoor, “Handwritten Signature Recognition: a Convolutional Neural Network Approach,†International Conference on Advanced Computation and Telecommunication (ICACAT). IEEE, p. (1-5), 2018. doi: 10.1109/icacat.2018.8933575.

W. Xiao and Y. Ding, “A Two-Stage Siamese Network Model for Offline Handwritten Signature Verification,†Symmetry (Basel), vol. 14, no. 6, p. 1216, 2022, doi: 10.3390/sym14061216.

P. D. Hung, P. S. Bach, B. T. Vinh, H. T. Nguyen, and V. T. Diep, “Offline Handwritten Signature Forgery Verification Using Deep Learning Methods,†Smart Trends in Computing and Communications: Proceedings of SmartCom. Springer Nature Singapore., p. (75-84), 2022. doi: 10.1007/978-981-16-9967-2_8.

A. L. Hagstrom, R. Stanikzai, J. Bigün, and F. Alonsoâ€Fernandez, “Writer Recognition Using Offline Handwritten Single Block Characters,†International Workshop on Biometrics and Forensics (IWBF). IEEE, p. (1-6), 2022. doi: 10.1109/iwbf55382.2022.9794466.

S. S. Harakannanavar, J. H, A. C. N, K. Prashanth, and P. Hudedavar, “Biometric Trait: Offline Signature Identification and Verification Based on Multi-modal Fusion Techniques,†Journal of Positive School Psychology, vol. 6, no. 4, pp. 2180–2191, 2022, [Online]. Available: https://journalppw.com/index.php/jpsp/article/view/3592

A. Jain, S. K. Singh, and K. P. Singh, “Handwritten Signature Verification Using Shallow Convolutional Neural Network,†Multimed Tools Appl, vol. 79, no. 27–28, pp. 19993–20018, 2020, doi: 10.1007/s11042-020-08728-6.

Y. Zhou, J. Zheng, H. Hu, and Y. Wang, “Handwritten Signature Verification Method Based on Improved Combined Features,†Applied Sciences, vol. 11, no. 13, p. 5867, 2021, doi: 10.3390/app11135867.

G. Li and H. Sato, “Sensing In-Air Signature Motions Using Smartwatch: A High-Precision Approach of Behavioral Authentication,†IEEE Access, vol. 10, pp. 57865–57879, 2022, doi: 10.1109/access.2022.3177905.

Y. Guo and H. Sato, “Smartwatch In-Air Signature Time Sequence Three-Dimensional Static Restoration Classification Based on Multiple Convolutional Neural Networks,†Applied Sciences, vol. 13, no. 6, p. 3958, 2023, doi: 10.3390/app13063958.

S. Franceschini, M. Ambrosanio, V. Pascazio, and F. Baselice, “Hand Gesture Signatures Acquisition and Processing by Means of a Novel Ultrasound System,†Bioengineering, vol. 10, no. 1, p. 36, 2023, doi: 10.3390/bioengineering10010036.

C. Sun, A. Shrivastava, S. Singh, and A. Gupta, “Revisiting Unreasonable Effectiveness of Data in Deep Learning Era,†Proceedings of the IEEE International Conference on Computer Vision. pp. 843–852, 2017.

Z. Y. Poo, C. Y. Ting, Y. P. Loh, and K. I. Ghauth, “Multi-Label Classification with Deep Learning for Retail Recommendation,†Journal of Informatics and Web Engineering, vol. 2, no. 2, pp. 218–232, 2023, doi: 10.33093/jiwe.2023.2.2.16.

L. Jia-Rou, K.-W. Ng, and Y.-J. Yoong, “Face and Facial Expressions Recognition System for Blind People Using ResNet50 Architecture and CNN,†Journal of Informatics and Web Engineering, vol. 2, no. 2, pp. 284–298, 2023, doi: 10.33093/jiwe.2023.2.2.20.

W. H. Khoh, Y. H. Pang, and H. Y. Yap, “In-air Hand Gesture Signature Recognition: an iHGS Database Acquisition Protocol,†F1000Res, p. 283, 2023, doi: 10.12688/f1000research.74134.2.

S. Ioffe and C. Szegedy, “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift,†International Conference on Machine Learning. pmlr, pp. 448–456, 2015.

W. H. Khoh, Y. H. Pang, and A. B. J. Teoh, “In-air hand gesture signature recognition system based on 3-dimensional imagery,†Multimed Tools Appl, vol. 78, no. 6, pp. 6913–6937, 2018, doi: 10.1007/s11042-018-6458-7.

J. C. Davis and A. F. Bobick, “The Representation and Recognition of Human Movement Using Temporal Templates,†Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE, p. (928-934), 2002. doi: 10.1109/cvpr.1997.609439.

Md. A. R. Ahad, J. K. Tan, H. Kim, and S. Ishikawa, “Motion History image: Its Variants and Applications,†Mach Vis Appl, vol. 23, no. 2, pp. 255–281, 2010, doi: 10.1007/s00138-010-0298-4.

C. Szegedy et al., “Going Deeper with Convolutions,†Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1–9, 2015. [Online]. Available: https://www.cv-foundation.org/openaccess/content_cvpr_2015/html/Szegedy_Going_Deeper_With_2015_CVPR_paper.html

A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks,†Commun ACM, vol. 60, no. 6, pp. 84–90, 2012, doi: 10.1145/3065386.

K. Simonyan and A. Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition,†arXiv.org. 2015. [Online]. Available: https://arxiv.org/abs/1409.1556

K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,†Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, pp. 770–778, 2016.