LFD Book Forum Exercise 4.6
 Register FAQ Calendar Mark Forums Read

#1
04-15-2016, 03:50 AM
 ntvy95 Member Join Date: Jan 2016 Posts: 37
Exercise 4.6

Hello, I have this answer for the Exercise 4.6 but I'm not sure if it's right?

Because for any , very small weights are still as powerful as large weights (all that matters is the accuracy of the calculations that computer being able to perform): That also means a hyperplane can be represented by many hypotheses, constraining the weights can reduce the number of hypotheses represents the same hyperplane. Hence soft-order constraint will be able to reduce the component while likely not compromising the component.

----------------------------------------

Edit: I have just remembered that the growth function has already taken care of the issue many hypotheses representing the same hyperplane (and this issue does not affect the component anyway (?)). So in this case the answer should be the hard-order constraint...? I'm really confused right now.
#2
11-08-2016, 03:09 PM
 CountVonCount Member Join Date: Oct 2016 Posts: 17
Re: Exercise 4.6

I have the same question. Can someone help here?

From my understanding having small weights is not perfect for sign(s), since this will lead to a signal that is often around 0 and thus a small change of just one input has a high chance to lead to a completely different output, if the sign changes.

So it would be better to have big weights, thus the signal is always pushed to the big number region and the sign is more stable.

But I maybe I'm just wrong here.
#3
11-09-2016, 06:32 AM
 magdon RPI Join Date: Aug 2009 Location: Troy, NY, USA. Posts: 592
Re: Exercise 4.6

Yes, the soft order constraint does not impact classification. Better regularize with the hard order constraint, or use the soft order constraint with the "regression for classification" algorithm.

Quote:
 Originally Posted by ntvy95 Hello, I have this answer for the Exercise 4.6 but I'm not sure if it's right? Because for any , very small weights are still as powerful as large weights (all that matters is the accuracy of the calculations that computer being able to perform): That also means a hyperplane can be represented by many hypotheses, constraining the weights can reduce the number of hypotheses represents the same hyperplane. Hence soft-order constraint will be able to reduce the component while likely not compromising the component. ---------------------------------------- Edit: I have just remembered that the growth function has already taken care of the issue many hypotheses representing the same hyperplane (and this issue does not affect the component anyway (?)). So in this case the answer should be the hard-order constraint...? I'm really confused right now.
__________________
Have faith in probability
#4
11-09-2016, 06:35 AM
 magdon RPI Join Date: Aug 2009 Location: Troy, NY, USA. Posts: 592
Re: Exercise 4.6

Correct again.

So let us differentiate between the theory of machine learning and its implementation on finite precision computers. In theory, if you have an infinite precision machine, then the size of the weights does not matter because it is a mathematical fact that, for positive ,

In finite precision, you typically want the weights to be around 1 and the inputs rescaled to be around 1 too (this is called input preprocessing and you can read about it in e-Chapter 9).

Quote:
 Originally Posted by CountVonCount I have the same question. Can someone help here? From my understanding having small weights is not perfect for sign(s), since this will lead to a signal that is often around 0 and thus a small change of just one input has a high chance to lead to a completely different output, if the sign changes. So it would be better to have big weights, thus the signal is always pushed to the big number region and the sign is more stable. But I maybe I'm just wrong here.
__________________
Have faith in probability
#5
11-09-2016, 10:00 AM
 CountVonCount Member Join Date: Oct 2016 Posts: 17
Re: Exercise 4.6

Thanks for this clarification. It helps a lot for understanding.
#6
11-10-2016, 06:44 AM
 ntvy95 Member Join Date: Jan 2016 Posts: 37
Re: Exercise 4.6

 Thread Tools Display Modes Linear Mode

 Posting Rules You may not post new threads You may not post replies You may not post attachments You may not edit your posts BB code is On Smilies are On [IMG] code is On HTML code is Off Forum Rules
 Forum Jump User Control Panel Private Messages Subscriptions Who's Online Search Forums Forums Home General     General Discussion of Machine Learning     Free Additional Material         Dynamic e-Chapters         Dynamic e-Appendices Course Discussions     Online LFD course         General comments on the course         Homework 1         Homework 2         Homework 3         Homework 4         Homework 5         Homework 6         Homework 7         Homework 8         The Final         Create New Homework Problems Book Feedback - Learning From Data     General comments on the book     Chapter 1 - The Learning Problem     Chapter 2 - Training versus Testing     Chapter 3 - The Linear Model     Chapter 4 - Overfitting     Chapter 5 - Three Learning Principles     e-Chapter 6 - Similarity Based Methods     e-Chapter 7 - Neural Networks     e-Chapter 8 - Support Vector Machines     e-Chapter 9 - Learning Aides     Appendix and Notation     e-Appendices

All times are GMT -7. The time now is 11:24 PM.