Databases
StackOverflow2013
Postsdbo
POSimple feedforward neural network does not behave as intended
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POSimple feedforward neural network does not behave as intended
    primarykey
    data
    text
    <p>I'm having a hard time figuring out why my feed forward artificial neural networks almost always fail to learn my simple "OR" perceptron:</p> <ul> <li>0 or 0 : 0 </li> <li>1 or 0 : 1 </li> <li>0 or 1 : 1 </li> <li>1 or 1 : 1</li> </ul> <p>I tried various network configurations : 1 or 2 hidden layers, 2 to 8 neurons on hidden layers, a bias (value = 1, connected to each hidden neuron and output neuron)</p> <p>For this example, the network looks like:</p> <pre><code>[1] [3] [2] [4] [6] [5] </code></pre> <ul> <li>input layer : [1] and [2]</li> <li>hidden layer : [3] and [4] and [6]</li> <li>output layer : [6]</li> </ul> <p><strong>Notations:</strong></p> <ul> <li>Neuron i : [i] </li> <li>Weight between [i] and [j] : w(i,j) </li> <li>[i] value : v(i)</li> <li>[i] error: e(i) </li> <li>sig(t) : 1/(1 + e^-t)</li> </ul> <p><strong>Weights initialization</strong></p> <p>Each weight is set to a random value between -0.5 and 0.5</p> <p><strong>Input</strong></p> <p>v(1) and v(2) are set with random values : 0 or 1</p> <p><strong>Propagate value (from left to right) :</strong> </p> <ul> <li>v(3) = sig(v(1) * w(1,3) + v(2) * w(2,3)) </li> <li>v(4) = sig(v(1) * w(1,4) + v(2) * w(2,4))</li> <li>v(5) = sig(v(1) * w(1,5) + v(2) * w(2,5))</li> <li>v(6) = sig(v(3) * w(3,6) + v(4) * w(4,6) + v(5) * w(5,6))</li> </ul> <p><strong>expected output : 1 if (v(1) or v(2)) is true, 0 otherwise</strong></p> <ul> <li>e(6) = (expected - v(6)) * v(6) * (1 - v(6))</li> </ul> <p><strong>Propagate error (from right to left)</strong></p> <ul> <li>e(3) = e(6) * w(3,6) * v(3) * (1 - v(3)) </li> <li>e(4) = e(6) * w(4,6) * v(4) * (1 - v(4)) </li> <li>e(5) = e(6) * w(5,6) * v(5) * (1 - v(5))</li> </ul> <p><strong>Update weights (learning rate = 1)</strong></p> <ul> <li>w(1,3) = w(1,3) + v(1) * e(3) </li> <li>w(1,4) = w(1,4) + v(1) * e(4) </li> <li>w(1,5) = w(1,5) + v(1) * e(5)</li> <li>w(2,3) = w(2,3) + v(2) * e(3) </li> <li>w(2,4) = w(2,4) + v(2) * e(4) </li> <li>w(2,5) = w(2,5) + v(2) * e(5)</li> <li>w(3,6) = w(3,6) + v(3) * e(6) </li> <li>w(4,6) = w(4,6) + v(4) * e(6) </li> <li>w(5,6) = w(5,6) + v(5) * e(6)</li> </ul> <p><strong>Do it 300 times and print each result:</strong></p> <ul> <li>if v(6) > 0.5, it’s a True </li> <li>if v(6) &lt; 0.5, it’s a False</li> <li>compare with expected output</li> </ul> <p>After a few epoch, the network almost always return v(6) > 0.9, even for v(1) = v(2) = 0</p> <p>Sometimes (roughly 1 out of 20 times), it works, the network learned correctly</p> <p>What am I doing wrong ?</p> <p><strong>EDIT:</strong> I found it !</p> <p>The culprits was :</p> <ul> <li>e(3) = e(6) * w(3,6) <strong>* v(3) * (1 - v(3))</strong> </li> <li>e(4) = e(6) * w(4,6) <strong>* v(4) * (1 - v(4))</strong> </li> <li>e(5) = e(6) * w(5,6) <strong>* v(5) * (1 - v(5))</strong></li> </ul> <p>Which should have been :</p> <ul> <li>e(3) = e(6) * w(3,6) </li> <li>e(4) = e(6) * w(4,6)</li> <li>e(5) = e(6) * w(5,6)</li> </ul>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USCyrille
    1. USCyrille
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. VO
      singulars
      1. POSimple feedforward neural network does not behave as intended
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POSimple feedforward neural network does not behave as intended
      1. USCyrille
      1. VTFavorite
    1. CO2 hidden layers? you only need a single neuron to represent an 'OR' (i.e. no hidden layer). So maybe the problem is, that the network is "too large" for this problem? (that will at least slow down convergence) Also your learning rate should not be equal to 1 but <1 (and decrease over time?)
      singulars
      1. POSimple feedforward neural network does not behave as intended
      1. USexample
    2. COI tried 2 hidden layers just to see what happen. And I tried to decrease learning rate over time too (with no better results). My guess is either my algorithm or my code have errors. Thank you anyway for the clarification.
      singulars
      1. POSimple feedforward neural network does not behave as intended
      1. USCyrille
    3. COI don't get it, `delta(hidden) = sum(deta(output) * weight(output, hidden)) * (value(hidden)) * (1-(value(hidden)) ` is the fixed formula, how could omitting the last two terms make it work???
      singulars
      1. POSimple feedforward neural network does not behave as intended
      1. USカオナシ
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload