
#1




Machine Learning and census models
Hello All:
I have the data for 14 venezuelan census that were done between 1873 and 2011. That is, I have N=14 data points. I am fitting this data to several different models (P is the population function)
So I have 14 data points and I'm estimating 2, 3 4 and even 5 parameters... Is this a ridiculously hopeless situation? By the way, the way I fit these models is by "running" them (dynamic simulation) using numerical differential equation solvers and fitting the parameters using nonlinear optimization. I don't know if this can be considered a data learning situation in some cases, like in the Generalized Logistic model, the search space is very complex and there are many many local minima where the optimization algorithms settle, in other cases the optimization algorithm converges into a single minimum error point. At any rate, the situation is that I have some data points and different models with different VC dimentions, hence I made the analogy with data learning... What I'd like to do is to be able to generalize and predict with a certain confidence level once I have a "good" model: what is the population carrying capacity the model predicts? when is the inflection point reached? What are the implications of that on the demographic transition stages the country goes through? etc. etc. I realize that there is no possible way to predict this given the possibility of wars, famine, epidemics that can drastically alter population dynamics and cannot be accounted for by any model. However, under the supposition that whatever the mechanisms of populations dynamics that any model implies that have been at work in the past will continue to be in effect in the future, can the data learning tools provide any way to validate such models? Or is the situation hopeless given the small N and large VC dimensions? 
#2




Re: Machine Learning and census models
My opinion is: Without enough data, it would be difficult for Machine Learning/Data Mining or even Human Intelligence to reach any conclusive statement about the process behind. Hope this helps.
__________________
When one teaches, two learn. 
#3




Re: Machine Learning and census models
How about this approach:
1. Don't look at the data! If you have looked at the data, find a machine learning expert who has not looked at the data and ask him to do it for you. [Unless you have some method of forgetting what you have seen, that is.] 2. Pick a learning method suited to the size of the data set and use leaveoneout crossvalidation to find the optimal hypothesis. The interesting question is what learning method in the second part. Something pretty general and regularized. 3. Bear in mind that extrapolation of nonstationary processes is not necessarily possible (the cross validation has an easy time of it, because most of the data points are internal). 
#4




Re: Machine Learning and census models
On reflection, I suspect that if the aim is extrapolation into the future a more principled alternative to function approximation with leaveoneout crossvalidation might be a variation where the in sample data for each run consists only of the data preceding the out of sample data point. (The reason is that as fitting intermediate points is easier, it could easily lead to overfitting).
Has anyone any views on this rather important general situation? 
#5




Re: Machine Learning and census models
Quote:
__________________
Where everyone thinks alike, no one thinks very much 
#6




Re: Machine Learning and census models
Quote:
However, with much more data than the 14 points described here, methods of the type I have researched in the past use two levels of design (similar to those discussed in your lectures and examples about crossvalidation) to select the class of model (but generally for hyperparameters, rather than choice of the order of a polynomial approximation). The input data may be consecutive values in the past or some other derived data. The predicted data may be some single value in the future. A data set consists of thousands of such data points of inputs and one output, constructed from a single data set. The aim is still to be able to predict the output from some new set of inputs. It is in the generation of a method to do this that my point arose (and I spotted jlaurentum's example is analogous). One approach looks at the several thousand data points as a sample from a distribution and applies cross validation followed by training with all of the data to generate a hypothesis for that distribution. This seems fine, but has an implicit assumption of stationarity. When intermediate points are being used to validate, future points are being used to generate the model. But the behaviour of the system in the future may depend on how it has behaved at intermediate points (this is how people make their decisions about what to do, and what their programs are using as data). Hence there is a subtle cheat going on here related to nonstationarity. Whether it is harmful or not, I am not yet sure. This is how I arrived at the alternative approach of only using validation data that is in the future of the data used to select hyperparameters, as well as the generation of the final model. This is something rather like cross validation, but different (as the data used in training is restricted). The choice of the training and validation window sizes is interesting: in both cases there is a compromise between wanting it to have lots of points, but to not extend over too much time (due to nonstationarity). It has some similarity to a method used by technical traders called walkforward optimisation which uses future out of sample errors to validate a method. This modified approach is not necessarily completely immune from nonstationarity. It relies on the models being used being sophisticated enough to capture the changing behaviour of the system as a whole, so may fail if this becomes not true at some time. I am not aware if there is much published on the way in which nonstationarity plays a role in this scenario. One thing that strikes me is the way that the selection of hyperparameters through crossvalidation (or the variant above) and the generation of a predictive hypothesis breaks the learning process up into two separate parts in quite a surprising way (even though a very neat and effective one). I wonder whether there are any other alternative ways to organise the overall learning process that might give good results? 
Thread Tools  
Display Modes  

