Design of a Low-area Digit Recognition Accelerator Using MNIST Database

Joonyub Kwon - Department of Semiconductor System Engineering, Sangmyung University, 31, Sangmyeongdae-gil, Dongnam-gu, Cheonan-si, Chungcheongnam-do, 31066, Republic of Korea
Sunhee Kim - Department of Semiconductor System Engineering, Sangmyung University, 31, Sangmyeongdae-gil, Dongnam-gu, Cheonan-si, Chungcheongnam-do, 31066, Republic of Korea


Citation Format:



DOI: http://dx.doi.org/10.30630/joiv.6.1.855

Abstract


Deep neural networks, which is a field of artificial intelligence, have been used in various fields. Deep learning is processed on high-performance GPUs or TPUs. It requires high cost as much as its good performance. Recently, as the demand for edge computing increases, many studies have been conducted to perform complex deep learning operations in a low-computing processor. Among them, a typical study is to lighten the deep learning network. In this paper, we propose a handwritten digit recognition hardware accelerator suitable for edge computing using MNIST database. After setting the correction rate for MNIST to 94% and performing network lighting processes, a hardware structure that can reduce the area of hardware and minimize memory access is proposed. Basically, the network is set as a two-layer fully connected network. The network is modeled with Python and lighten while checking the performance. Network parameters, weighs and biases, are quantized. The pixel number and bit number of MNIST input data are also reduced. The number of MAC units and the processing order of the hardware accelerator are determined so that there is no not used MACs while performing the MAC operations in parallel. It is designed with Verilog HDL and its functions are checked in Modelsim. And then it is implemented in Xilinx Zynq ZC-702 to verify the operations. The designed number recognition accelerator is expected to be widely used in edge devices by reducing the area and memory access.

Keywords


MNIST; accelerator; digit recognition; edge computing; fully-connected network.

Full Text:

PDF

References


S. Li, W. Song, L. Fang, Y. Chen, P. Ghamisi and J. A. Benediktsson, “Deep Learning for Hyperspectral Image Classification: An Overview,†IEEE Transactions on Geoscience and Remote Sensing, vol. 57, no. 9, pp. 6690-6709, Sept. 2019, doi: 10.1109/TGRS.2019.2907932.

S.L. Oh, Y. Hagiwara, U. Raghavendra, R. Yuvaraj, N. Arunkumar, M. Murugappan and U. R. Acharya, “A deep learning approach for Parkinson’s disease diagnosis from EEG signals,†Neural Comput & Applic.. vol. 32, pp. 10927–10933, 2020. 10.1007/s00521-018-3689-5.

L. Jiao and J. Zhao, “A Survey on the New Generation of Deep Learning in Image Processing,†IEEE Access, vol. 7, pp. 172231-172263, 2019. 10.1109/ACCESS.2019.2956508.

N. Justesen, P. Bontrager, J. Togelius and S. Risi, “Deep Learning for Video Game Playing,†IEEE Transactions on Games, vol. 12, no. 1, pp. 1-20, March 2020. 10.1109/TG.2019.2896986.

T. Liang, J. Glossner, L. Wang, S. Shi, and X. Zhang, “Pruning and Quantization for Deep Neural Network Acceleration: A Survey,†Neurocomputing, vol. 461, pp. 370-403, Oct. 2021, 10.1016/j.neucom.2021.07.045.

V. Lebedev, V. Lempitsky, “Speeding-up convolutional neural networks: A survey,†Bulletin of the Polish Academy of Sciences. Technical Science, vol. 66, no. 6, pp. 799-811, 2018, 10.24425/bpas.2018.125927.

M. D. Zeiler, and R. Fergus. “Stochastic pooling for regualization of deep convolutional neural networks,†arXiv preprint arXiv:1301.3557, 2013, 10.48550/arXiv.1301.3557.

J. Y. Wu, C. Yu, S. W. Fu, C. T. Liu, S. Y. Chien and Y. Tsao, “Increasing Compactness of Deep Learning Based Speech Enhancement Models With Parameter Pruning and Quantization Techniques,†IEEE Signal Processing Letters, vol. 26, no. 12, pp. 1887-1891, Dec. 2019. 10.1109/LSP.2019.2951950.

J. Guo, W. Liu, W. Wang, J. Han, R. Li, Y. Lu, S. Hu, “Accelerating Distributed Deep Learning By Adaptive Gradient Quantization,†in Proc. ICASSP, Barcelona, Spain, 2020, pp. 1603-1607, doi: 10.1109/ICASSP40776.2020.9054164.

C. Wang, L. Gong, X. Li, and X. Zhou, “A Ubiquitous Machine Learning Accelerator With Automatic Parallelization on FPGA,†IEEE Transactions on Parallel and Distributed Systems, vol. 31, no. 10, pp. 2346-2359, 1 Oct. 2020. 10.1109/TPDS.2020.2990924.

Y. Toyama, K. Yoshioka, K. Ban, S. Maya, A. Sai and K. Onizuka, “An 8 Bit 12.4 TOPS/W Phase-Domain MAC Circuit for Energy-Constrained Deep Learning Accelerators,†IEEE Journal of Solid-State Circuits, vol. 54, no. 10, pp. 2730-2742, Oct. 2019. 10.1109/JSSC.2019.2926649.

Y. Wang, Y. Wang, H. Li and X. Li, “An Efficient Deep Learning Accelerator Architecture for Compressed Video Analysis,†IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems. pp. 1-1, 2021. 10.1109/TCAD.2021.3120076.

J. Chen and X. Ran, “Deep Learning With Edge Computing: A Review,†Proceedings of the IEEE, vol. 107, no. 8, pp. 1655-1674, Aug. 2019. 10.1109/JPROC.2019.2921977.

K. Cao, Y. Liu, G. Meng and Q. Sun, “An Overview on Edge Computing Research,†IEEE Access, vol. 8, pp. 85714-85728, 2020. 10.1109/ACCESS.2020.2991734.

W. Z. Khan, E. Ahmed, S. Hakak, I. Yaqoob, and A. Ahmed, “Edge computing: A survey,†Future Generation Computer Systems, vol. 97, pp. 219-235, 2019. 10.1016/j.future.2019.02.050.

S. W. Yang, “Efficient Deep Learning on Limited System Resources in FPGAs Performance Comparison on Floating Points,†M.S. thesis, Dept. Computer & Information Technology, Korea Univ., Seoul, Korea, 2019.

A. Iwata, Y. Yoshida, S. Matsuda, Y. Sato, and N. Suzumura, “An artificial neural network accelerator using general purpose 24 bits floating point digital signal processors,†in IJCNN, Washington, DC, USA, vol. 2, 1989, pp.171–175, doi: 10.1109/IJCNN.1989.118695.

J. Civit-Masot, F. Luna-Perejón, S. Vicente-Díaz, J. M. Rodríguez Corral and A. Civit, “TPU Cloud-Based Generalized U-Net for Eye Fundus Image Segmentation,†IEEE Access, vol. 7, pp. 142379-142387, 2019. 10.1109/ACCESS.2019.2944692.

R. Murillo, A. A. D. Barrio, and G. Botella, “Deep PeNSieve: A deep learning framework based on the posit number system,†Digital Signal Processing, vol. 102, 102762, 2020. 10.1016/j.dsp.2020.102762.

Q. H. Vo, N. Linh Le, F. Asim, L. W. Kim and C. S. Hong, “A Deep Learning Accelerator Based on a Streaming Architecture for Binary Neural Networks,†IEEE Access, vol. 10, pp. 21141-21159, 2022. 10.1109/ACCESS.2022.3151916.

Y. Toyama, K. Yoshioka, K. Ban, S. Maya, A. Sai and K. Onizuka, “An 8 Bit 12.4 TOPS/W Phase-Domain MAC Circuit for Energy-Constrained Deep Learning Accelerators,†IEEE Journal of Solid-State Circuits, vol. 54, no. 10, pp. 2730-2742, Oct. 2019. 10.1109/JSSC.2019.2926649.

H. F. Langroudi, Z. Carmichael, and D. Kudithipudi, “Deep Learning Training on the Edge with Low-Precision Posits,†arXiv preprint arXiv.1907.13216, 2019. 10.48550/arXiv.1907.13216.

H. W. Son, D. Y. Lee, and H. W. Kim, “Compact CNN Accelerator Chip Design with Optimized MAC And Pooling Layers,†Journal of the Korea Institute of Information and Communication Engineering, vol. 25, no. 9, pp. 1158-1165, Sept. 2021, 10.6109/JKIICE.2021.25.9.1158.

S. F. Hsiao, K. C. Chen, C. C. Lin, H. J. Chang, and B. C. Tsai, “Design of a Sparsity-Aware Reconfigurable Deep Learning Accelerator Supporting Various Types of Operations,†IEEE Journal on Emerging and Selected Topics in Circuits and Systems, vol. 10, no. 3, pp. 376-387, Sept. 2020. 10.1109/JETCAS.2020.3015238.

L. Kang, H. Li, X. Li, and H. Zheng, “Design of Convolution Operation Accelerator based on FPGA,†in Proc. Int. Conf. MLBDBI, Taiyuan, China, 2020, pp. 80-84, doi:10.1109/MLBDBI51377.2020.00021.

Y. LeCun, C. Cortes, and C. J. C. Burges, "The Mnist Database of handwritten digits," [Online]. Available: http://yann.lecun.com/exdb/mnist/

L. Wan, M. Zeiler, S. Zhang, Y. L. Cun, and R. Fergus, “Regularization of Neural Networks using DropConnect,†in Proc. the International Conference on Machine Learning, PMLR, Atlanta, Georgia, USA, 2013, pp. 1058-1066.

Siham Tabik, Ricardo F. Alvear-Sandoval, María M. Ruiz, José-Luis Sancho-Gómez, Aníbal R. Figueiras-Vidal, Francisco Herrera, “MNIST-NET10: A heterogeneous deep networks fusion based on the degree of certainty to reach 0.1% error rate. Ensembles overview and proposal,†Information Fusion, vol. 62, pp. 73-80, Oct. 2020. 10.1016/j.inffus.2020.04.002.

Matuzas77, “MNIST-0.17,†[Online]. Available: https://github.com/Matuzas77/MNIST-0.17

S. S. Kadam, A. C. Adamuthe, and A. B. Patil, “CNN Model for Image Classification on MNIST and Fashion-MNIST Dataset,†Journal of Scientific Research, vol. 64, no. 2, pp. 374-384, 2020. 10.37398/JSR.2020.640251.

R. F. Alvear-Sandoval, J. L. Sancho-Gómez, and A. R. Figueiras-Vidal, “On improving CNNs performance: The case of MNIST,†Information Fusion, vol. 52, pp. 106-109, Dec. 2019. 10.1016/j.inffus.2018.12.005.

A. Baldominos, Y. Saez, and P. Isasi, “A Survey of Handwritten Character Recognition with MNIST and EMNIST,†Appl. Sci., vol. 9, no. 15, 3169, Aug. 2019, 10.3390/app9153169.

A. Velichko A, “Neural Network for Low-Memory IoT Devices and MNIST Image Recognition Using Kernels Based on Logistic Map,†Electronics, vol. 9, no. 9, 1432, 2020. 10.3390/electronics9091432

S. S, Mor, S. Solanki, S. Gupta, S. Dhingra, M. Jain, and R. Saxena, “Handwritten Text Recognition: with Deep Learning and Android,†IJEAT, vol. 8, no. 3S, pp. 819-825, Feb. 2019.

S. Himanshu, “Activation Functions : Sigmoid, tanh, ReLU, Leaky ReLU, PReLU, ELU, Threshold ReLU and Softmax basics for Neural Networks and Deep Learning,†Jan. 19, 2019. [Online]. Available: https://himanshuxd.medium.com/activation-functions-sigmoid-relu-leaky-relu-and-softmax-basics-for-neural-networks-and-deep-8d9c70eed91e