What is deep learning?
What is is and why it is so interesting? Basically, deep learning is an umbrella term for a whole set of advanced algorithms based usually on neural networks that consist of multiple hidden layers. More formally, according to Li Deng and Dong Yu, who published an excellent publication about deep learning (Deep Learning: Methods and Applications), it can be defined as:
A class of machine learning techniques that exploit many layers of non-linear information processing for supervised or unsupervised feature extraction and transformation, and for pattern analysis and classification.
A sub-field within machine learning that is based on algorithms for learning multiple levels of representation in order to model complex relationships among data. Higher-level features and concepts are thus defined in terms of lower-level ones, and such a hierarchy of features is called a deep architecture.
There are many other different definitions of deep learning, however, these two concepts are common among all of them:
- multiple layers or stages of non linear processing
- methods for supervised or unsupervised learning of feature representation at successively higher, more abstract layers
One of the major advantages of deep learning algorithms is that, unlike traditional machine learning techinques, they do not rely on feature engineering to obtain good results. Deep learning systems can automatically detect and generate more complex, high-level features from raw data sources. Let’s take a look at a simple example. Given raw input of image pixes intensities a deep learning net first learns to discover basic patterns such as edges, then objects parts and finally whole objects.
Depending on how deep learning techniques are intended to use, they can be divded into three categories:
- deep architectures for supervised learning
- deep architectures for unsupervised learning
- deep architectures for hybrid learning (supervised + unsupervised)
Deep learning algorithms include deep neural networks, deep belief networks, convolutional neural networks, autoencoders and many more. If you are interested in detailed information about different types of neural networks, there is an excellent course on Coursera by Geoff Hinton. There is also a nice, introductory article by Ivan Vasilev.
Since 2006 deep learning algorithms have been applied to a diverse range of problems such as computer vision, audio and video processing, information retrieval, natural language processing and understanding, biometrics and robotics. The real impact of deep learning in the industry started in 2010 when Geoff Hinton and Li Deng started working on apply deep learning in speech recognition tasks. One of the major features of deep learning – ability to find features in raw data has been explored to build deep neural networks for speech recognition. Large-scale automatic speech recognition was the first and the most convincing case of practical use of deep learning in recent years, embraced by both industry and academic. What is more, all major speech recognition systems (Google Now, Apple Siri, Microsoft Cortana, Baidu, Skype, Xbox, Skype Translator) are based on deep learning algorithms.
The wide-spreading success in speech recognition achieved in 2011 was shortly followed by large-scale image recognition. In 2012 team of Geoff Hinton and his students won the ImageNet competition providing a solution using convolutional neural networks. They significantly outperformed other participants, proving that deep learning can be used in a large-scale for image classification problems. Today, solutions based on convolutional neural networks are used for plenty of tasks – from image recognition, video analysis and even natural language processing. DeepDream, a system created by Google which uses convolutional neural networks, which was initially developed for ImageNet challenge, is able to detect faces and other patters and in a result create hallucinogenic appearances in processed images.
As you can see innocent looking image of a ladybird turned into somewhat disturbing result. The neural net detected some eyes, spiders, fish and even a head of a dog and dozens of undefined creatures.
Deep learning brings impressive solutions and advantages in many areas. More and more sophisticated algorithms are being developed and applied to create state-of-the-art solutions. Top companies like Google, Facebook invest in deep learning by hiring the most prominent researchers and improve they products by developing and enhancing existing deep learning techniques. In 2014 Google acquired British startup DeepMind that developed a system capable of learning how to play Atari video games using only raw pixels as data input.
There are plenty of interesting resources about deep learning and neural networks.
- Deep Learning article on Wikipedia
- Deep Learning Boo
- Neural Networks and Deep Learning
- DEEP LEARNING: Methods and Applications
- Neural Networks for Machine Learning
- Andrew Ng: Deep Learning, Self-Taught Learning and Unsupervised Feature Learning
- Yann LeCun – The Unreasonable Effectiveness of Deep Learning
- A Deep Learning Tutorial: From Perceptrons to Deep Networks
In the next article I will focus on more practical approach to deep learning. I will show how to use popular deep learning libraries in order to apply deep learning techniques to your problems.