Clarification on HW6-Q8

TonySuarez · #1 08-15-2012, 11:39 PM

In Q8 it is asked the "...closest to the total number of operations required in a single iteration of backpropagation (using SGD on one data point)".

My question is if we have to account only the backpropagation step or, instead, one complete iteration of the backpropagation algorithm, which includes the forward propagation, the backpropagation and the SGD updating steps.

Thanks in advance.

TS

yaser · #2 08-15-2012, 11:48 PM

Quote:

Originally Posted by TonySuarez

In Q8 it is asked the "...closest to the total number of operations required in a single iteration of backpropagation (using SGD on one data point)".

My question is if we have to account only the backpropagation step or, instead, one complete iteration of the backpropagation algorithm, which includes the forward propagation, the backpropagation and the SGD updating steps.

Forward, backward, and update.

dvs79 · #3 08-16-2012, 05:49 AM

Did I understand right that dimensions for input and hidden layer are given without constant term? (so, that in input layer, e.g., we have 5 +1 (const) nodes).

yaser · #4 08-16-2012, 05:53 AM

Quote:

Originally Posted by dvs79

Did I understand right that dimensions for input and hidden layer are given without constant term? (so, that in input layer, e.g., we have 5 +1 (const) nodes).

Correct, as the convention has the index

going from

to $d^{(l-1)}$ .

TonySuarez · #5 08-16-2012, 07:49 AM

Thank you very much Professor for answering "on-the-fly" (as always).
TS

invis · #6 08-17-2012, 03:05 AM

Only product of the form $... w_{ij}^{(l)} \delta_j^{(l)}...$ count as operations.

On backward we have: $\delta_i^{(l-1)} = (1-(x_i^{(l-1)})^2) \sum_{j=1}^{d^{(l)}}w_{ij}^{(l)} \delta_j^{(l)}$

$d^{(2)}=1$ and $d^{(1)}=3$ and +1 (constant) for layer (1). So we have the same 4 operations for 2-1 layer, and obviously 18 operations for 1-0 layer. Or we dont need to compute $\delta$ for constants ?

Thread Tools
Show Printable Version Email this Page
Display Modes
Switch to Linear Mode Hybrid Mode Switch to Threaded Mode