Solving a Backpropagation Problem

What is the problem

You’ll be given,

• the nodes, connections and weights of a neural network
• the activation function of the hidden layers.
  (usually is a relu)
• the activation function of the o/p layer. (usually is a sigmoid/softmax)
• the error function.
• the learning rate $$\eta =0.1$$

… and asked to perform backpropagation on it.

Before you begin

1. draw the network, leaving at lease a line’s space above every neuron
   (to later write in it the net and out of the neuron)
2. add the target value of every output node next to it, and the input value of every input node also next to it.
   
3. perform forward propagation to fill the net and out values for every node.
   
4. compute the derivative formula for the activation function of both the o/p layer and the hidden layers.
   
5. compute the derivative formula for the error function.

The strategy would change if u were to use batch gradient descent vs the stochastic one.

Strategy

Using Stochastic Gradient Descent

Starting with the output layer’s neurons and moving back, do the following,

1. calculate the neuron’s delta (${\delta }_{O1}$)
   
2. update the weight connecting it to every neuron in the previous layer.
   

Do this for every neuron within every layer…

---

Connections