_
Hello,
in the following I will present You a way to develop a neural network with just two input neurones and one output neurone
that simulates an AND Gate.
In my opinion this is a good start to learn the algorithms of backpropagation and its way it works.
I will use_ C# as programming languages but it is easy to port to any other language like C++ or Java.
Let's start!
First let me explain You what a neurone in the context of machine learning is.
Short answer: A neurone is a unit that takes values from previous neurones, sums them and returns the output value which is passed through an activation function ( https://en.wikipedia.org/wiki/Artificial_neuron#Types_of_transfer_functions).
Neural networks are seperated in layers.
We will use an one layer perceptron ( one input layer and one output layer).
For operations like XOR we would need an additional hidden layer ( "The middle layer is called a hidden layer, since the neurons in this layer are neither inputs nor outputs." http://neuralnetworksanddeeplearning.com/chap1.html)
because XOR is not "linear separable". ( for a imho good explanation please visit https://www.quora.com/What-is-XOR-problem-in-neural-networks).
However let's return to our AND Gatter.
What is an AND Gatter?
Given is a set of bits
{0, 0}
{1, 0}
{0, 1}
{1, 1}
If we pass these tuples into an AND Gatter
it would return
{0}
{0}
{0}
{1}
This means the AND Gatter checks whether two bits are 1. If it is it returns 1 otherweise 0.
This means our neural network has to return 1 if the input is {1,1} otherwise 0.
Let's start with the programming.
We create a class called IActivation which is an interface that represents the activation function for a neurone:
After this we can create our first two class that holds the activation function:
and the linear activation function for the input layer because we want the raw values:
Our generalized neurone:
Some create 2 classes of neurones and name them InputNeurone and WorkingNeurone but with this approach the neurones are directly charaterized by its activation function.
The connection class:
Every neurone has a List of connection.
A connection between two neurones is established by passing neurone A ( from neurone) and neurone B ( to neurone) to the constructor.
Furthermore a weight is given.
By adjusting the weights the neural network can archieve "learning".
This is what neural networks make so effective: The output values of a neural network to an input X can be manipulated to the desired value.
We will use this to "teach" the neural network to recognize pattern ( in our case the pattern of bits ) and give automatically the correct answer _.
The method "CalculateDelta(double requestedValue)" calculates the error of the specific neurone.
Its error is measured by the partial derivative of the quadratic cost function.
A cost function gives information about how "good" the network is.
The goal of backpropagation is now finding the minimum of the loss( or cost) function by adjusting the weights in order to make the loss less.
Last but not least we create a class SimpleNetwork that manages all important functions ( like forwarding values and backpropagating):
In the beginning all values ( weights of the two inputs and bias) are arbitrarily set.
In the backwardpass of the backpropagation process the weights and bias are going to be adjusted.
The backpropagation method iterates through all training datas until either the pre defined accuracy or the maximum amount of epochs is reached.
In addition to that in each epoch the error is calculated by the quadratic cost function( https://stats.stackexchange.com/questions/154879/a-list-of-cost-functions-used-in-neural-networks-alongside-applications).
In order to get the accuracy we calculate accuracy = 1 - error ( The less the error the more accurate the network is.)
And this is how we use it:
If we run the application we got following results:
_
Seems to have worked.
Best regards!
Post scriptum: Backpropagation learned from this video ( german):
_