Clarification on HW6-Q8

[TonySuarez]

In Q8 it is asked the "...closest to the total number of operations required in a single iteration of backpropagation (using SGD on one data point)".

My question is if we have to account only the backpropagation step or, instead, one complete iteration of the backpropagation algorithm, which includes the forward propagation, the backpropagation and the SGD updating steps.

Thanks in advance.

TS

[yaser]

Originally Posted by TonySuarez

In Q8 it is asked the "...closest to the total number of operations required in a single iteration of backpropagation (using SGD on one data point)".

My question is if we have to account only the backpropagation step or, instead, one complete iteration of the backpropagation algorithm, which includes the forward propagation, the backpropagation and the SGD updating steps.

Forward, backward, and update.

[dvs79]

Did I understand right that dimensions for input and hidden layer are given without constant term? (so, that in input layer, e.g., we have 5 +1 (const) nodes).

[yaser]

Originally Posted by dvs79

Did I understand right that dimensions for input and hidden layer are given without constant term? (so, that in input layer, e.g., we have 5 +1 (const) nodes).

Correct, as the convention has the index going from to .

[TonySuarez]

Thank you very much Professor for answering "on-the-fly" (as always).
TS

[invis]

Only product of the form count as operations.

On backward we have:

and and +1 (constant) for layer (1). So we have the same 4 operations for 2-1 layer, and obviously 18 operations for 1-0 layer. Or we dont need to compute for constants ?

[dvs79]

yes, we don't need to:

1. compute delta for the output (because it doesn't need any of the operations, counted as operation in this certain task)

2. compute deltas for constants (because they're constants)

3. compute deltas for input (because they're just features (x), and delta is a derivative of the error with respect to s)

So for computing deltas you only need 3 operations.

[invis]

Thank you very much ! Enlgish isnt my native language and its hard to me to catch 100% information from video. Thanks for clarifying !

[marek]

Originally Posted by dvs79

yes, we don't need to:

1. compute delta for the output (because it doesn't need any of the operations, counted as operation in this certain task)

2. compute deltas for constants (because they're constants)

3. compute deltas for input (because they're just features (x), and delta is a derivative of the error with respect to s)

So for computing deltas you only need 3 operations.

Thank you! Your answer preempted one of my questions. Specifically about the calculation of the final layer delta, as technically none of it seems to count as an operation under our definition.