machine learning - Neural Network Diverging instead of converging -


i have implemented neural network (using cuda) 2 layers. (2 neurons per layer). i'm trying make learn 2 simple quadratic polynomial functions using backpropagation.

but instead of converging, diverging (the output becoming infinity)

here more details i've tried:

  • i had set initial weights 0, since diverging have randomized initial weights
  • i read neural network might diverge if learning rate high reduced learning rate 0.000001
  • the 2 functions trying add are: 3 * + 7 * j+9 , j*j + i*i + 24 (i giving layer i , j input)
  • i had implemented single layer , approximate polynomial functions better
  • i thinking of implementing momentum in network i'm not sure learn
  • i using linear (as in no) activation function
  • there oscillation in beginning output starts diverging moment of weights become greater 1

i have checked , rechecked code there doesn't seem kind of issue it.

so here's question: going wrong here?

any pointer appreciated.

  1. if problem trying solve of classification type, try 3 layer network (3 enough accordingly kolmogorov) connections inputs , b hidden node c (c = a*wa + b*wb) represent line in ab space. line divides correct , incorrect half-spaces. connections hidden layer ouput, put hidden layer values in correlation each other giving desired output.

  2. depending on data, error function may hair comb, implementing momentum should help. keeping learning rate @ 1 proved optimum me.

  3. your training sessions stuck in local minima every once in while, network training consist of few subsequent sessions. if session exceeds max iterations or amplitude high, or error high - session has failed, start another.

  4. at beginning of each, reinitialize weights random (-0.5 - +0.5) values.

  5. it helps chart error descent. "aha!" factor.


Comments

Post a Comment

Popular posts from this blog

basic authentication with http post params android -

i doing basic authentication passing username , password , using basicnamevaluepair sending post params response service. my method: public stringbuilder callservicehttppost(string username, string password, string type) { // create new httpclient , post header httpclient httpclient = new defaulthttpclient(); httppost httppost = new httppost(webservice + type); httpresponse response = null; stringbuilder total = new stringbuilder(); try { url url = new url(webservice + type); /*string base64encodedcredentials = base64.encodetostring((username + ":" + password).getbytes(), base64.url_safe | base64.no_wrap);*/ string base64encodedcredentials = "basic " + base64.encodetostring( (username + ":" + password).getbytes(), base64.no_wrap); httppost.setheader("auth...
Read more

vb.net - Virtual Keyboard commands -

i wondering difference between virtual these keyboard commands is: keyeventf_extendedkey , keyeventf_keyup is. everywhere have looked gives me description based off integers , not want know each of them does. you've tagged question vb.net, these have nothing @ vb.net. they're constants defined in windows header files, use win32 api functions. as far difference, can't tell looking @ values. individual values not particularly important, that's why named identifiers used. what's important used , documentation functions tells mean. the first one, keyeventf_extendedkey , used keybdinput structure (which used along e.g. sendinput function) pass information synthesized keyboard input. if flag used, means scan code should interpreted extended key. technically, means scan code preceded prefix byte value 224 (&he0 in hexadecimal notation). the second one, keyeventf_keyup , 1 of flags available use structure. means key being released (going up)...
Read more

android - Inheriting from Theme.AppCompat* -

currently working on migrating actionbar in support libraries. trying migrate old themes inherit theme.appcompat.light.darkactionbar isn't going smoothly. it fine if apply theme in manifest such: <activity android:name="com.fitsby.loginactivity" android:screenorientation="portrait" android:theme="@style/theme.appcompat.light.darkactionbar" > </activity> but runtime error, stating loginactivity(subclass of actionbaractivity) must have theme inherits theme.appcompat, when following: in styles.xml: <style name="apptheme" parent="@style/theme.appcompat.light.darkactionbar"> <item name="android:typeface">sans</item> </style> and in manifest: <activity android:name="com.fitsby.loginactivity" android:screenorientation="portrait" android:theme="@style/apptheme" > </activity> any ideas why happening? not...
Read more