DDL is available today in version 4 of PowerAI, and according to IBM, it contains implementations at various stages of development, for Caffe, Tensorflow, and Torch. Accuracy of the results is another issue contributing to the time factor, as the system needs to be taught multiple times in order to gain the desired results. As was pointed out by Hunter in her blog, scaling a deep learning problem across more GPUs is made much more hard as these processors get faster, since communication between them and the rest of the system struggles to keep pace as the computational power of the graphics chips increase.
“A$3 s GPUs get much faster, they learn much faster, and they have to share their learning with all of the other GPUs at a rate that isn’t possible with conventional software”.
In other words, bottlenecks get in the way.
IBM Research-Distributed Deep Learning-made a big stride in deep learning this week, achieving a record for image recognition accuracy of 33.8%. “In addition, the communication latency plays an important role in massive scaling of GPUs (over 100)”.
To achieve these results IBM used a clustering technology that acts as an overseer of the many processors that are involved. This was necessary due to the use of multiple servers with GPUs, with IBM fellow Hilary Hunt explaining that the more GPUs there are, or if they are of a higher quality, the learning time can actually be slower as they have more to “talk” about. “Specifically, our team wrote software and algorithms that automate and optimize the parallelization of this very large and complex computing task across hundreds of GPU accelerators attached to dozens of servers”.
“My team in IBM Research has been focused on reducing these training times for large models with large data sets”, Hunter wrote. This bested the previous best scaling of 89 percent demonstrated by Facebook AI Research using smaller learning models and data sets, which reduced complexity.
The research tackles one of the major challenges of deploying deep learning: Large neural networks and large datasets help deep learning thrive but also lead to longer training times. Training large-scale, deep learning-based AI models can take days or weeks.
IBM has beaten the record for artificial intelligence (AI) image recognition by developing distributed deep learning (DDL) software.
What’s more, IBM’s system managed to achieve this in seven hours, while the process that allowed Microsoft to set the previous record took 10 days to complete. The NVIDIA V100 GPUs, which are just entering the market now, will offer 120 teraflops of mixed single/half precision performance, further challenging the ability of these deep learning applications to scale efficiently.
“We expect that by making this DDL feature available to the AI community, we will see many more higher accuracy runs as others leverage the power of clusters for AI model training”, Hunter said.