The number of neural network (NN) types is growing at an impressive rate. We will look at each type in the article and provide short descriptions.If you need to refresh your memory about some basic staff, related to the neural network, read this article
Feed-forward neural networks and perceptrons
Very straightforward, feed-forward neural networks and perceptrons (FFNN) transmit information from input to output. The cells in one layer are not connected, while neighboring layers are usually fully connected. The most straightforward neural network has two input cells and one output cell and can be used as a model of logic gates.
Auto Encoder
Auto Encoder (AE) is somewhat similar to feed-forward neural networks and perceptrons in that it is a different way of using the principles at work rather than a fundamentally different architecture. The basic idea is to automatically encode (in the sense of compressing, not encrypting) information. The network itself is shaped like an hourglass, with hidden layers smaller than the input and output. The neural network can be trained by backpropagation of error, feeding the input data and setting loss function to minimize the difference among the input and output.
Variational AE
Have a similar architecture to autoencoders, but train them differently: by approximating the probability distribution of the input samples. In this, they take their cue from Boltzmann machines. Nevertheless, they rely on Bayesian mathematics with probabilistic inference and independence, which are intuitive but challenging to implement. To summarize, we can say that this network considers the effects of neurons.
Denoising AE
These are autoencoders in which the input data are fed in a noisy state. We calculate the error and the output data compared to the noisy data. By doing this, the network learns to pay attention to the large properties since the smaller ones can change along with the noise.
Sparse AE
In a sense, the opposite of a regular autoencoder. Instead of training the network to display information in a smaller “volume” of nodes, we increase the number of nodes. Instead of tapering to the center, the network swells there. Networks of this type help deal with small properties of a data set. If you train a network as a regular auto-encoder, nothing good will come out of it. That’s why a special sparsity filter, which passes only particular errors, is fed in addition to input data.
Markov Chain
These are the predecessors of Boltzmann machines (BM) and Hopfield networks (HN). Their meaning can be explained as follows: what are my chances of getting to one of the following nodes if I am in this one? Each next state depends only on the previous one. Although Markov chains are not Hopfield networks, they are pretty similar. Also, Markov chains are not necessarily fully connected.
Hopfield Network
It is a neural network with a symmetric connection matrix. During training, each input becomes a hidden node. To train the network: the values of neurons are set according to the desired pattern, the weights are calculated, and they don’t change later.
The Boltzmann Machine
The Boltzmann machine (BM) is very similar to the Hopfield network, but with some neurons marked as input and some marked as hidden. The input neurons then become output neurons. The Boltzmann machine is a stochastic network. Learning takes place by the method of error backpropagation or by the algorithm of comparative divergence. The learning process is very similar to that of the Hopfield network.
Restricted BM
Surprisingly similar to the Boltzmann machine and, hence, to the Hopfield network. The only difference is its boundedness. In it, neurons of the same type are not connected.
It is trained like FFNN, but with one nuance: instead of forwarding data transmission and backward error propagation, you must transmit data first in the forward direction.
Deep Belief Network
It is the name given to the type of architecture in which a network consists of several interconnected Restricted BMs or Variational AEs. Such networks are trained block by block, with each block needing only to be able to encode the previous one. This technique is called “greedy learning,” which consists of selecting local optimal solutions that do not guarantee an optimal final result.
It can display data as a probabilistic model. With teacherless learning, the stabilized model can be used to generate new data.
Deconvolutional Network
Sometimes called inverse graphical networks. Imagine that you feed the word “cat” to the network, and it generates pictures of cats. Also, it can be combined with FFNNs.
Convolutional neural networks and Deep convolutional neural networks
Convolutional neural networks (CNN) and Deep convolutional neural networks (DCNN) are very different from other types of networks. They are typically used for image processing, less commonly for audio. A typical use of CNNs is to classify images: if there is a cat in the picture, the network will say “cat,” and if there is a dog, it will say “dog.” Such networks usually use a “scanner” that does not parse all the data.