View Single Post
Old 03-16-2013, 09:47 PM
rk1000 rk1000 is offline
Join Date: Jul 2012
Posts: 10
Default Re: Neural Network predictions converging to one value


The training data points do have different target outputs.

After training from a mini-batch of the training set and adjusting the weights, when I try to predict the outputs of the same training mini-batch, it produces identical (incorrect) outputs when it has undergone some training. Initially i.e. at the beginning of training, these values are very far wrong and also different but many are the same, but eventually after some training, all the predictions on the training set start to converge, and they get closer to the mean of the outputs of the latest training mini-batch.

I get the same output value if I then run the network on a validation set. When I continue training from the next mini-batch and then test with that mini-batch and then a validation set, the behavior is the same, but the predicted value changes after each training mini-batch.

I hope I have answered your question.

Initially, I had conjectured that maybe, the network is completely biased and is just providing a fixed output from the weight of the bias term and no contribution from the other weights, and so I removed the bias term altogether. But the behavior stayed the same, and I brought the bias term back.
Reply With Quote