Re: Question 9  minimum # of weights
I think I see your point (as long as you are not assuming there are any inputs to the constant bias nodes), but it is clear that if only biases were connected to each layer, the network would not be connected in a practical sense (as there is no information passing between layers) or in a topological sense. You know the answer. :)

Re: Question 9  minimum # of weights
In the lecture on neural networks it is mentioned that the number of weights works as a reference for the VC dimension of the network. Linking to this question, is there any guidance towards how to construct a neural net? I am thinking about the balance between the number of hidden layers and the number of units per layer. My intuition is that working with units near to equally distributed across the layers increases the VC dimension, so more expressiveness against larger generalasation error? In practice one would then decide based on a generalisation error analysis?

Re: Question 9  minimum # of weights
Quote:

Re: Question 9  minimum # of weights
I believe I recall reading that the range of functions which can be approximated to any given accuracy with multilayer networks is the same as the range achievable with networks with just 2 hidden layers. However, networks with one hidden layer are limited to approximating a more restricted (but also rather general) range of functions (which, on checking, I find consists of continuous functions on compact subsets of )
Of course this doesn't preclude networks with a greater number of hidden layers being better in some other definable sense. [Thinking of natural neural networks such as those in our brains, it is natural for these to be very deep, using multiple levels of processing feeding into each other]. Regarding design of neural networks, I've experimented with them on several occasions over several years and have applied rules of thumb for design. As well as generally limiting hidden layers to 2, one idea concerns how much data you need to justify using a certain complexity of neural network. While it is normal to use validation to stop training when overfitting occurs, I suspect there is no advantage to having lots of neurons if stopping occurs too early to make good use of them. One practical formula is: where is the tolerable error. Sorry I can't locate where this came from: perhaps someone else knows? There is also theoretical work on estimating the VCdimensions of NNs, such as "VapnikChervonenkis Dimension of Neural Nets" by Peter L. Bartlett. 
Re: Question 9  minimum # of weights
Many thanks for your answers, that's really helpful and interesting. Have googled the reference Elroch posted and found quite a few references, going through them at the moment!

Re: Question 9  minimum # of weights
You might find this reference interesting as well (especially as regards Yaser's comment) :)
http://yann.lecun.com/exdb/mnist/index.html 
All times are GMT 7. The time now is 12:56 PM. 
Powered by vBulletin® Version 3.8.3
Copyright ©2000  2021, Jelsoft Enterprises Ltd.
The contents of this forum are to be used ONLY by readers of the Learning From Data book by Yaser S. AbuMostafa, Malik MagdonIsmail, and HsuanTien Lin, and participants in the Learning From Data MOOC by Yaser S. AbuMostafa. No part of these contents is to be communicated or made accessible to ANY other person or entity.