LFD Book Forum  

Go Back   LFD Book Forum > Course Discussions > Online LFD course > Homework 5

Reply
 
Thread Tools Display Modes
  #1  
Old 05-04-2013, 05:09 AM
arcticblue arcticblue is offline
Member
 
Join Date: Apr 2013
Posts: 17
Default Basic logistic regression question

From lecture 9 page 23 of the slides there is an algorithm of how to implement logistic regression. In step 3 it explains how to compute the gradient. Is the E-in value actually a vector or is it a single number? If it's a single number then the weight would be the same for every value in the weight vector so it seems like E-in is a vector. Is my understanding correct?

And if it's a vector then I'm a little unclear on how to compute the values. Each training point has two values x1 and x2 and an outcome y. So to calculate E-in do I just use x1 and weight1 to find the first value and then use x2 and weight2 to find the second value?

Hopefully the above makes sense, I seem to be struggling with something that seems like it should be pretty simple.
Reply With Quote
  #2  
Old 05-04-2013, 11:10 AM
yaser's Avatar
yaser yaser is offline
Caltech
 
Join Date: Aug 2009
Location: Pasadena, California, USA
Posts: 1,477
Default Re: Basic logistic regression question

Quote:
Originally Posted by arcticblue View Post
From lecture 9 page 23 of the slides there is an algorithm of how to implement logistic regression. In step 3 it explains how to compute the gradient. Is the E-in value actually a vector or is it a single number? If it's a single number then the weight would be the same for every value in the weight vector so it seems like E-in is a vector. Is my understanding correct?
E_{\rm in} is a scalar quantity whose value depends on the entire vector of weights {\bf w}. Step 3 computes the gradient, which is a vector of derivatives of E_{\rm in} with respect to each of those weights, so each component of the gradient vector is the derivative with respect to a different weight
__________________
Where everyone thinks alike, no one thinks very much
Reply With Quote
  #3  
Old 05-04-2013, 09:27 PM
arcticblue arcticblue is offline
Member
 
Join Date: Apr 2013
Posts: 17
Default Re: Basic logistic regression question

Okay I think I see how that works but I'm still struggling to understand Q8. In the question I set the weights to 0. Then the first time through the loop I will calculate the gradient of E_in using the formula in step 3. Because w is all zeros the denominator will end up as 1+ e^0 == 2. The numerator can at most be +/-1/ So the biggest change in gradient is +/-0.5 for each weight.

Then in step 4 I update the weights
w(1) = w0 - learningRate*gradient.
w(1) = 0,0,0 - 0.01(0.5,0.5,0.5)
w(1) = (0.005,0.005,0.005)

Now the question states stop the algorithm when ||w(t-1) and w(t)|| < 0.01. So:
sqrt((0-0.005)^2+(0-0.005)^2+(0-0.005)^2) = 0.008
So based on the values above the algorithm will stop after the first iteration because the difference in weights is < 0.01.

Have I misunderstood the gradient of E_in formula? Or am I calculating my error incorrectly? I've tried using batch gradient descent and see the above results (I have 100 data points but the error still ends up less than 0.01.) I've also tried stochastic gradient descent and have similar problems.

I've watched lecture 9 a couple of times now and seem to understand how the algorithm works but I guess my understanding isn't complete.

Any suggestions would be most appreciated.
Reply With Quote
  #4  
Old 05-04-2013, 10:38 PM
yaser's Avatar
yaser yaser is offline
Caltech
 
Join Date: Aug 2009
Location: Pasadena, California, USA
Posts: 1,477
Default Re: Basic logistic regression question

Quote:
Originally Posted by arcticblue View Post
Now the question states stop the algorithm when ||w(t-1) and w(t)|| < 0.01.
The values are calculated at the end of an epoch per the preamble to Question 8, so you need to run through all the examples first to get to the end of the epoch.
__________________
Where everyone thinks alike, no one thinks very much
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 04:35 PM.


Powered by vBulletin® Version 3.8.3
Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.
The contents of this forum are to be used ONLY by readers of the Learning From Data book by Yaser S. Abu-Mostafa, Malik Magdon-Ismail, and Hsuan-Tien Lin, and participants in the Learning From Data MOOC by Yaser S. Abu-Mostafa. No part of these contents is to be communicated or made accessible to ANY other person or entity.