ANN Series 9 - Multilayer Perceptron(MLP) and Multilayer Artificial Neural Network(Multi-layered ANN)

The terms "Multilayer Perceptron" (MLP) and "Multilayer Artificial Neural Network" (Multi-layered ANN) are often used interchangeably to describe a specific type of neural network architecture. Both refer to a class of artificial neural networks that contain one input layer, one or more hidden layers, and one output layer. 

The key characteristics that define MLPs and multi-layered ANNs include:

1. Multiple Layers: 

Both MLPs and multi-layered ANNs are characterized by having multiple layers of neurons (also known as nodes). These include the input layer that receives the data, one or more hidden layers that process the data, and the output layer that produces the final prediction or classification.

2. Non-linear Activation Functions: 

In both architectures, neurons in the hidden layers and sometimes in the output layer apply non-linear activation functions to their inputs. These functions enable the network to learn complex patterns and relationships in the data that linear models cannot. Common activation functions include sigmoid, tanh, and ReLU (Rectified Linear Unit).

3. Fully Connected Layers: 

MLPs and multi-layered ANNs typically consist of fully connected layers, meaning each neuron in one layer is connected to all neurons in the subsequent layer. This dense connectivity allows the network to capture complex interactions between input features.

4. Backpropagation for Training: 

Both use backpropagation, a powerful algorithm for training neural networks, in combination with an optimization technique (such as stochastic gradient descent) to adjust the weights of the connections between neurons. This training process minimizes a predefined loss function, improving the model's accuracy over time.

5. Universal Approximation Capability: 

Thanks to their structure and non-linear activation functions, MLPs and multi-layered ANNs are capable of approximating any continuous function given sufficient neurons in the hidden layers, a property known as the Universal Approximation Theorem.

While "Multilayer Perceptron" is a term specifically used to describe a feedforward neural network with one or more hidden layers, "Multilayer Artificial Neural Network" is a broader term that can encompass MLPs as well as other types of networks with multiple layers. However, in many contexts, when people refer to multi-layered ANNs, they are often thinking of MLPs due to their commonality and foundational role in neural network architectures.

The following content shows a Two-layered Perceptron and how forward pass is made on it.

The following content shows a Three-layered Perceptron and how forward pass is made on it.

Python Code:

The following code shows how to create an MLP using Pytorch.

ml-course/Pytorch_hidden nodes.ipynb at main · lovelynrose/ml-course (github.com)

Try changing the hyperparameters like

1. Number of hidden layers

2. Number of hidden nodes in each layer

3. Learning rate

4. Number of epochs

5. Optimization technique

Youtube Link:

Detailed explanation for the code can be found in the following video.

Multi- Layered Perceptron for Regression

In the realm of regression, single-layered ANNs, often referred to as linear regression models when equipped with linear activation functions, are adept at modeling relationships between variables that can be described by a straight line or a linear equation. These models are powerful for problems where the relationship between the input variables and the target variable is linear or approximately linear. However, real-world data often exhibit complex, non-linear relationships that single-layered networks, due to their simplicity, struggle to capture accurately.

The introduction of non-linear activation functions in single-layered networks does allow for a degree of non-linear modeling capability. Functions like the sigmoid or tanh can enable these networks to bend and twist the decision boundary or regression line, offering a better fit than purely linear models for certain less complex non-linear patterns. Yet, their capacity to encapsulate the full spectrum of non-linear complexities in data is limited.

The leap to Multilayer Perceptrons (MLPs) represents a significant advancement in the ability to model complex non-linear relationships for regression tasks. MLPs, with their multiple layers of neurons and non-linear activation functions, can approximate any continuous function to a high degree of accuracy.

Multi- Layered Perceptron for Classification

So far, we have discussed single-layered Artificial Neural Networks (ANNs), such as the perceptron, which are primarily capable of learning linear decision boundaries or solving problems that can be linearly separated. These networks, when equipped with a linear activation function or no activation function at all, cannot model complex non-linear relationships inherent in many real-world datasets.

However, by introducing non-linear activation functions, even single-layer networks can start to address less complex non-linear problems to some extent, especially when the data can be transformed into a linearly separable space through such non-linear mappings. Common non-linear activation functions include the sigmoid, tanh, and ReLU (Rectified Linear Unit).

Multilayer Perceptrons (MLPs) are a type of multi-layered ANN. Unlike their single-layer counterparts, MLPs are composed of one input layer, multiple hidden layers, and one output layer, with non-linear activation functions applied at each layer. This architecture enables MLPs to learn highly complex non-linear relationships and decision boundaries. The key to their power lies in the depth of the network—having multiple layers (depth) and non-linear activations allows MLPs to approximate virtually any continuous function, as per the Universal Approximation Theorem.