How I did Deep Learning Research with a CPU Laptop?
Recently I published a paper titled “Sparse Low Rank Factorization for Deep Neural Network Compression" in Neurocomputing Journal.
Here is a short story of how this work was developed in my office laptop (Windows 10, 8GB RAM, Intel i5, 512 GB HDD). Though we have a NVIDIA Supercomputer DGX V100 (128 GB of GPU, 512 GB RAM, 8 TB HDD, 2 Intel Xeon CPU) for AI and Deep Learning research at our Bennett University, I had to work on a laptop. But why? I was traveling during every weekends of past 1 year conducting Deep Learning workshops at various Engineering Institutions in India. I get free time mostly during flight travel and stay at hotels. Internet becomes a headache when you want to connect to your server from remote locations (and that too in travel). So, I had to work on my laptop. The objective of the research is to compress fully connected layer of a trained deep CNN model using low-rank matrix factorization approach such as SVD. The research needs experiments with standard benchmark models such as AlexNet and VGG16. I was pretty sure that decomposition of large fully connected layer matrix and further testing the model on ImageNet dataset in a laptop is nearly impossible. So, I chose to work on model and dataset that most of us don’t focus much. As I am a 90s kid, I trained a Lenet5 model on MNIST dataset (both introduced in 1990s). Here are the advantages in working with Lenet5 and MNIST. Lenet5 model has a fully connected layer weight matrix of size 400 x 120. Testing set of MNIST consist of 10,000 images of size 28 x 28. I had to train the model once and save it to the disk. Testing of the model will take only few minutes. That's it. It cannot get anymore simpler. But here is the difficult thing I did not know about Lenet5 model. The uncompressed model achieves testing accuracy 98.49% which is very hard to maintain after compressing with SVD. I realized that compressing a smaller model is much more difficult than compressing the larger models since smaller models don’t have much redundancy as the larger ones do. Within few weeks, I developed a sparse variation of the SVD which achieved better compression rate without compromising the testing accuracy of Lenet5. Once the development was over, experiments with larger models on standard datasets were boosted up with our NVIDIA DGX V100 supercomputer. My faith is restored once again: “Resource availability should never be the stopping factor for any work”. Thanks for reading. Cheers! #ArtificialIntelligence #DeepLearning #MachineLearning #DataScience #DataAnalytics #Dataset #Research #Leadingindiaai #CSEBennett #BennettUniversity #Engineering #EngineeringEducation #Elsevier #NVIDIA #DGX #GPU #CPU #CNN #SVD #ImageNet #VGG16
Bye from Dr. Sridhar Swaminathan
Published on 08 September 2021
Share
About the Author
