Heterogeneous Computing Applied to Deep Learning

Deep Learning is a new activity field that it will offer many opportunities for emerging markets and services and will revolutionize almost every segment of the society. With Deep Learning techniques, it is possible to improve Artificial Neural Networks (ANN) and Machine Learning (ML) algorithms and it will be possible to provide a high precision tool to address classification and prediction problems such as, for instance, speech synthesis or pattern recognition. Depth Learning covers a wide range of activities: medical, multimedia, finance, or advertising, among others. The results are applied to the most varied systems, such as clusters, mobile phones, or even smart sensors. A preliminary market analysis revealed that the Deep Learning market would increase from $109 million in 2015 to $10.4 billion in 2024 [1]. However, Depth Learning requires intensive processing of large databases on high-performance servers. Thus, for companies such as Intel, IBM, Amazon or Google, energy consumption becomes a serious issue. Their servers and databases are now considering the adoption at the architectural level of low power alternatives, as fast as powerful central computers, multi-core or GPUs (Graphics Processing Unit). Indeed, Deep Learning implementations have a high computational cost because most techniques use ANN with several layers [2]. This feature makes it difficult to apply Deep Learning algorithms to many emerging fields such as Mining Massive Dataset (MMD) or Bioinformatics.

New architectures such as Reconfigurable Computing (RC), Graphics Processing Unit (GPU) and High-Performance Computing (HPC) can improve the performance of Deep Learning algorithms. RC enables the development of customizable hardware architectures tailored to the algorithms, unlike the traditional CPU or general purpose hardware. With the advent of customizable hardware (using field-programmable gate arrays - FPGAs), algorithms can be parallelized and optimized at the gate level to speed up operations, reaching up to 500x according to what is presented in the literature [3, 4, 5]. Due to the increased use of FPGA-based reconfigurable computing in HPC, in addition to consumer, automotive or military electronics, the FPGA market is expected to grow to $ 12.1 billion by 2024. The expansion towards new application domains and the hardware cost reduction are the main reasons for the growth of this market [6]. GPUs have also been intensively used to speed up ANN algorithms and, in several works, researchers have found peaks of speedup of about 1000x compared to other hardware platforms. As with FPGAs, applications using GPUs will generate a multi-billion dollars market in the coming years. However, many studies have shown that the advantages and disadvantages of FPGAs and GPUs in targeting ANN algorithms have motivated migration to a heterogeneous computation where the HPC + GPU + FPGA combined attributes offer better results [7]. Today, companies such as Intel, with the new neural compute engine Movidius Myriad X vision processing unit (VPU) [8], promotes custom architectures for Artificial intelligence bringing processing speed and low power consumption to embedded systems.

[1] “Deep Learning Software Market to Surpass $10 Billion by 2024”, tractica.com. https://www.tractica.com/newsroom/press-releases/deep-learning-software-market-to-surpass-10-billion-by-2024/ (September 13, 2017).

[2] M. Gheisari, G. Wang and M. Z. A. Bhuiyan, "A Survey on Deep Learning in Big Data," 2017 IEEE International Conference on Computational Science and Engineering (CSE) and IEEE International Conference on Embedded and Ubiquitous Computing (EUC), Guangzhou, 2017, pp. 173-180.

[3] de Souza, A.C.D.; Fernandes, M.A.C. Parallel Fixed Point Implementation of a Radial Basis Function Network in an FPGA. Sensors 2014, 14, 18223-18243.

[4] Kevin M. Irick, Michael DeBole, Vijaykrishnan Narayanan, and Aman Gayasen. A hardware efficient support vector machine architecture for FPGA. 16th International Symposium on Field-Programmable Custom Computing Machines, 2008.

[5] Nalini C. Iyer and Sagarika Mandal. Implementation of secure hash algorithm-1 using FPGA. International Journal of Information and Computation Technology, 3(8):757–764, 2013.

[6] “FPGA Market to Reach $12.1B by 2024”, epsnews.com. https://epsnews.com/2017/09/11/fpga-market-reach-12-1b-2024/ (September 13, 2017).

[7] A. Reisizadeh, S. Prakash, R. Pedarsani, and S. Avestimehr, "Coded computation over heterogeneous clusters," 2017 IEEE International Symposium on Information Theory (ISIT), Aachen, 2017, pp. 2408-2412.

[8] Remi El-Ouazzane. Intel, Introducing Myriad X: Unleashing AI at the Edge. https://newsroom.intel.com/editorials/introducing-myriad-x-unleashing-ai-at-the-edge/. August 28.