Howto Calculate An Error's Partial Derivative in ANN

Question

This is a follow-on question from this post I made, "Multilayer Perceptron (Neural Network) for Time Series Prediction", a few months back.

I'm constructing a feed-forward artificial neural network, using resilient propagation training. At the moment, I'm trying to implement an individual neuron input's weight update algorithm. For the life of me, I can't seem to find a clear and straightforward answer on how to calculate the partial derivative of the error for a given weight. The only thing I can find on the web, is the fact that a neuron's weight update is a function of $\frac{dE}{dW}$ (cf. the original Paper [p. 2 & 3], or this one [p. 4]).

However none of these papers actually outlines how to calculate this.

I understand the concept of a partial derivative in a mathematical sense. And I assume that the current neuron input's weight change calculation is the operation at hand, while all other neuron input values are held constant.

So for each of these neurons below, I calculate each inputs' individual error by taking a total error ( -0.3963277746392987 ), that's been multiplied by that neuron input's weight (each :calculated-error is the sum of the individual inputs' error).

For both neurons, what would be the weight change for each input?

Here is my data structure:

    :input-layer                                                                                                                                                                                                     
     ({:calculated-error -1.0991814559154283,                                                                                                                                                                         
       :calculated-value 0.9908633780805893,                                                                                                                                                                            
       :inputs                                                                                                                                                                                                          
       ({:error -0.07709937922001887,                                                                                                                                                                                   
         :calculated 0.4377023624017325,                                                                                                                                                                                  
         :key :avolume,                                                                                                                                                                                                   
         :value 2.25,                                                                                                                                                                                                     
         :weight 0.19453438328965889,                                                                                                                                                                                     
         :bias 0}                                                                                                                                                                                                         
        {:error -0.19625185888745333,                                                                                                                                                                                    
         :calculated 1.4855269156904067,                                                                                                                                                                                  
         :key :bvolume,                                                                                                                                                                                                   
         :value 3.0,                                                                                                                                                                                                      
         :weight 0.4951756385634689,                                                                                                                                                                                      
         :bias 0}                                                                                                                                                                                                         
        {:error -0.3072203938672436,                                                                                                                                                                                     
         :calculated 1.0261589301119642,                                                                                                                                                                                  
         :key :ask,                                                                                                                                                                                                       
         :value 1.32379,                                                                                                                                                                                                  
         :weight 0.7751674586693994,                                                                                                                                                                                      
         :bias 0}                                                                                                                                                                                                         
        {:error -0.36920086975057054,                                                                                                                                                                                    
         :calculated 1.2332848282147972,                                                                                                                                                                                  
         :key :bid,                                                                                                                                                                                                       
         :value 1.3239,                                                                                                                                                                                                   
         :weight 0.9315543683169403,                                                                                                                                                                                      
         :bias 0}                                                                                                                                                                                                         
        {:error -0.14940895419014188,                                                                                                                                                                                    
         :calculated 0.5036129016361643,                                                                                                                                                                                  
         :key :time,                                                                                                                                                                                                      
         :value 1.335902400676,                                                                                                                                                                                           
         :weight 0.37698330460468044,                                                                                                                                                                                     
         :bias 0}),                                                                                                                                                                                                       
       :id "583c10bfdbd326ba525bda5d13a0a894b947ffc"},
      ...)

    :output-layer                                                                                                                                                                                                   
    ({:calculated-error -1.1139741279964241,                                                                                                                                                                        
      :calculated-value 0.9275622253607013,                                                                                                                                                                         
      :inputs                                                                                                                                                                                                       
      ({:error -0.2016795955938916,                                                                                                                                                                                 
        :calculated 0.48962608882549025,                                                                                                                                                                            
        :input-id "583c10bfdbd326ba525bda5d13a0a894b947ffb",                                                                                                                                                        
        :weight 0.5088707087900713,                                                                                                                                                                                 
        :bias 0}                                                                                                                                                                                                    
       {:error -0.15359996014735702,                                                                                                                                                                                
        :calculated 0.3095962076691644,                                                                                                                                                                             
        :input-id "583c10bfdbd326ba525bda5d13a0a894b947ffa",                                                                                                                                                        
        :weight 0.38755790024342773,                                                                                                                                                                                
        :bias 0}                                                                                                                                                                                                    
       {:error -0.11659507401745359

        :calculated 0.23938733624830652,                                                                                                                                                                            
        :input-id "583c10bfdbd326ba525bda5d13a0a894b947ff9",                                                                                                                                                        
        :weight 0.2941885012312543,                                                                                                                                                                                 
        :bias 0}                                                                                                                                                                                                    
       {:error -0.2784739949663631,                                                                                                                                                                                 
        :calculated 0.6681581686752845,                                                                                                                                                                             
        :input-id "583c10bfdbd326ba525bda5d13a0a894b947ff8",                                                                                                                                                        
        :weight 0.7026355778870271,                                                                                                                                                                                 
        :bias 0}                                                                                                                                                                                                    
       {:error -0.36362550327135884,                                                                                                                                                                                
        :calculated 0.8430641676611533,                                                                                                                                                                             
        :input-id "583c10bfdbd326ba525bda5d13a0a894b947ff7",                                                                                                                                                        
        :weight 0.9174868039523537,                                                                                                                                                                                 
        :bias 0}),                                                                                                                                                                                                  
      :id "583c10bfdbd326ba525bda5d13a0a894b947ff6"})

Thanks in advance

Answer 1

It's a bit unclear to me what you're trying to do, and maybe a better place to ask your question is Stats.SE but I would encourage you to go and have a look at this online class on machine learning which provides an implementation of the backpropagation algorithm.

You can either register or hit preview and go to the NN:Learning chapter, but I would recommend you to register and do the programming exercise on NN. Maybe this is not exactly what you are trying to do, but I think you might find a basic example of how to compute the partial derivative of the error term.

Answer 2

The calculation of the gradient depends on the activation functions of your neurons (and, more specifically, their derivatives). Moreover, the gradient of any individual weight depends on weights and activations appearing later in the network -- that's the key idea behind the backpropagation algorithm, which takes advantage of that fact to compute gradients. You can find a very detailed description of backprop in this paper. Note that the examples use sigmoid activation functions, but the method is applicable to any differentiable activation.

Note also that the principal difference between backprop and rprop is that backprop requires the value of the weight gradient and rprop only requires its sign. Though rprop may seem simpler in that regard, I advise you to make sure you completely understand backprop first.