1
$\begingroup$

This is a follow-on question from this post I made, "Multilayer Perceptron (Neural Network) for Time Series Prediction", a few months back.

I'm constructing a feed-forward artificial neural network, using resilient propagation training. At the moment, I'm trying to implement an individual neuron input's weight update algorithm. For the life of me, I can't seem to find a clear and straightforward answer on how to calculate the partial derivative of the error for a given weight. The only thing I can find on the web, is the fact that a neuron's weight update is a function of $\frac{dE}{dW}$ (cf. the original Paper [p. 2 & 3], or this one [p. 4]).

However none of these papers actually outlines how to calculate this.

I understand the concept of a partial derivative in a mathematical sense. And I assume that the current neuron input's weight change calculation is the operation at hand, while all other neuron input values are held constant.

So for each of these neurons below, I calculate each inputs' individual error by taking a total error ( -0.3963277746392987 ), that's been multiplied by that neuron input's weight (each :calculated-error is the sum of the individual inputs' error).

For both neurons, what would be the weight change for each input?

Here is my data structure:


    :input-layer                                                                                                                                                                                                     
     ({:calculated-error -1.0991814559154283,                                                                                                                                                                         
       :calculated-value 0.9908633780805893,                                                                                                                                                                            
       :inputs                                                                                                                                                                                                          
       ({:error -0.07709937922001887,                                                                                                                                                                                   
         :calculated 0.4377023624017325,                                                                                                                                                                                  
         :key :avolume,                                                                                                                                                                                                   
         :value 2.25,                                                                                                                                                                                                     
         :weight 0.19453438328965889,                                                                                                                                                                                     
         :bias 0}                                                                                                                                                                                                         
        {:error -0.19625185888745333,                                                                                                                                                                                    
         :calculated 1.4855269156904067,                                                                                                                                                                                  
         :key :bvolume,                                                                                                                                                                                                   
         :value 3.0,                                                                                                                                                                                                      
         :weight 0.4951756385634689,                                                                                                                                                                                      
         :bias 0}                                                                                                                                                                                                         
        {:error -0.3072203938672436,                                                                                                                                                                                     
         :calculated 1.0261589301119642,                                                                                                                                                                                  
         :key :ask,                                                                                                                                                                                                       
         :value 1.32379,                                                                                                                                                                                                  
         :weight 0.7751674586693994,                                                                                                                                                                                      
         :bias 0}                                                                                                                                                                                                         
        {:error -0.36920086975057054,                                                                                                                                                                                    
         :calculated 1.2332848282147972,                                                                                                                                                                                  
         :key :bid,                                                                                                                                                                                                       
         :value 1.3239,                                                                                                                                                                                                   
         :weight 0.9315543683169403,                                                                                                                                                                                      
         :bias 0}                                                                                                                                                                                                         
        {:error -0.14940895419014188,                                                                                                                                                                                    
         :calculated 0.5036129016361643,                                                                                                                                                                                  
         :key :time,                                                                                                                                                                                                      
         :value 1.335902400676,                                                                                                                                                                                           
         :weight 0.37698330460468044,                                                                                                                                                                                     
         :bias 0}),                                                                                                                                                                                                       
       :id "583c10bfdbd326ba525bda5d13a0a894b947ffc"},
      ...)

    :output-layer                                                                                                                                                                                                   
    ({:calculated-error -1.1139741279964241,                                                                                                                                                                        
      :calculated-value 0.9275622253607013,                                                                                                                                                                         
      :inputs                                                                                                                                                                                                       
      ({:error -0.2016795955938916,                                                                                                                                                                                 
        :calculated 0.48962608882549025,                                                                                                                                                                            
        :input-id "583c10bfdbd326ba525bda5d13a0a894b947ffb",                                                                                                                                                        
        :weight 0.5088707087900713,                                                                                                                                                                                 
        :bias 0}                                                                                                                                                                                                    
       {:error -0.15359996014735702,                                                                                                                                                                                
        :calculated 0.3095962076691644,                                                                                                                                                                             
        :input-id "583c10bfdbd326ba525bda5d13a0a894b947ffa",                                                                                                                                                        
        :weight 0.38755790024342773,                                                                                                                                                                                
        :bias 0}                                                                                                                                                                                                    
       {:error -0.11659507401745359

        :calculated 0.23938733624830652,                                                                                                                                                                            
        :input-id "583c10bfdbd326ba525bda5d13a0a894b947ff9",                                                                                                                                                        
        :weight 0.2941885012312543,                                                                                                                                                                                 
        :bias 0}                                                                                                                                                                                                    
       {:error -0.2784739949663631,                                                                                                                                                                                 
        :calculated 0.6681581686752845,                                                                                                                                                                             
        :input-id "583c10bfdbd326ba525bda5d13a0a894b947ff8",                                                                                                                                                        
        :weight 0.7026355778870271,                                                                                                                                                                                 
        :bias 0}                                                                                                                                                                                                    
       {:error -0.36362550327135884,                                                                                                                                                                                
        :calculated 0.8430641676611533,                                                                                                                                                                             
        :input-id "583c10bfdbd326ba525bda5d13a0a894b947ff7",                                                                                                                                                        
        :weight 0.9174868039523537,                                                                                                                                                                                 
        :bias 0}),                                                                                                                                                                                                  
      :id "583c10bfdbd326ba525bda5d13a0a894b947ff6"})

Thanks in advance

$\endgroup$

2 Answers 2

2
$\begingroup$

It's a bit unclear to me what you're trying to do, and maybe a better place to ask your question is Stats.SE but I would encourage you to go and have a look at this online class on machine learning which provides an implementation of the backpropagation algorithm.

You can either register or hit preview and go to the NN:Learning chapter, but I would recommend you to register and do the programming exercise on NN. Maybe this is not exactly what you are trying to do, but I think you might find a basic example of how to compute the partial derivative of the error term.

$\endgroup$
2
  • $\begingroup$ Thanks for this. Andrew Ng is actually the guy whose videos I first started following, when I was getting into AI. But this course seems to emphasize using NNs for classification problems (discreet values). I want to compute a continuous value in a time series. But I will dig further into this series. Thanks again. $\endgroup$ Oct 5, 2012 at 3:25
  • $\begingroup$ There is another class specifically for NN if you look on the site. $\endgroup$
    – SRKX
    Oct 5, 2012 at 8:50
0
$\begingroup$

The calculation of the gradient depends on the activation functions of your neurons (and, more specifically, their derivatives). Moreover, the gradient of any individual weight depends on weights and activations appearing later in the network -- that's the key idea behind the backpropagation algorithm, which takes advantage of that fact to compute gradients. You can find a very detailed description of backprop in this paper. Note that the examples use sigmoid activation functions, but the method is applicable to any differentiable activation.

Note also that the principal difference between backprop and rprop is that backprop requires the value of the weight gradient and rprop only requires its sign. Though rprop may seem simpler in that regard, I advise you to make sure you completely understand backprop first.

$\endgroup$
6
  • $\begingroup$ I conceptually understand what you mean by "the gradient of any individual weight depends on weights and activations appearing later in the network". But I don't understand how that bears out when calculating weight changes. I've been using this paper as a guide to backprop. Anyways, I'll definitely go through the paper you provided. Thanks. $\endgroup$ Oct 5, 2012 at 3:33
  • $\begingroup$ Ahh, I'm getting what you mean by that statement. Starting to sink in slowly :) $\endgroup$ Oct 5, 2012 at 3:59
  • $\begingroup$ Hi @Nutritioustim, sorry if I wasn't clear -- an example will help. Say you're building a 2-layer neural network. Now suppose all the weights in your second layer are 0. Then the gradient for all the weights in your first layer must be 0, because no matter what you set them out, their output is going to be multiplied by 0 in the second layer, eliminating their impact. That's why backprop is such an important algorithm -- it takes the error back through each layer. Note that from a mathematic perspective, it's basically just a chain rule application. $\endgroup$
    – jlowin
    Oct 6, 2012 at 12:10
  • $\begingroup$ In your reference, note how the "error signal" at each neuron is the weighted sum of the error signals in the following layer. That error signal gets multiplied by the neuron's derivative and its input to find the weight update -- therefore the total gradient (i.e. the weight update) for that neuron is dependent on results in later layers. $\endgroup$
    – jlowin
    Oct 6, 2012 at 12:14
  • $\begingroup$ Yeah, I'm starting to get the jist of from the paper you provided - it's a doozey. I'm on p.16, "Steps of the algorithm". I'm sure I'll have 1 or 2 more Qs. It really comes down to the implementation details. As I'm trying to use this to build a neural network (see here) $\endgroup$ Oct 8, 2012 at 2:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.