I'd like to share two pieces of thoughts on your view of linear regression.

(0) While linear regression can be described as one-step learning, calculating the pseudo inverse of

(which is

by

) generally contains multiple steps in the order of

. So whether to call it one-step learning depends on how you view the process.

(1) While using the pseudo inverse is arguably the most common way of doing linear regression since the early years of Statistics, indeed it is not the only way. Applying gradient decent on linear regression, as you describe, can also work and would allow you to enjoy something similar to PLA (or logistic regression with gradient decent) --- making the line iteratively approach the optimal one.