LFD Book Forum

LFD Book Forum (http://book.caltech.edu/bookforum/index.php)
-   Chapter 4 - Overfitting (http://book.caltech.edu/bookforum/forumdisplay.php?f=111)
-   -   Does a test set exist? (http://book.caltech.edu/bookforum/showthread.php?t=9566)

gverhoev 08-10-2021 12:51 PM

Does a test set exist?
 
In his recent lecture on testing and validation, Professor Magdon-Ismail said that a test said is used in a fictitious world. Does that mean that the term/concept "test set" does in reality not exist, as one always splits a data set in training and validation sets? And as soon as data are split, they influence the final hypothesis g because there are fewer data points to train. So can one even use the concept of a test set? Or is there a rule when one could use the term test set, also in a real-world situation? Many thanks for your insights!

htlin 08-11-2021 04:36 AM

Re: Does a test set exist?
 
In my experience, a one-use-only test set usually does not exist in the real applications, given that "one-use-only" is arguably too luxurious in any application. When multiple tests are conducted on the same test set, the set becomes closer to a "validation" set than a test set. But anyway, IMHO those are just naming issues. The important thing is to know that every use on a set can cause some sort of "contamination."

Hope this helps.


All times are GMT -7. The time now is 09:44 AM.

Powered by vBulletin® Version 3.8.3
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.
The contents of this forum are to be used ONLY by readers of the Learning From Data book by Yaser S. Abu-Mostafa, Malik Magdon-Ismail, and Hsuan-Tien Lin, and participants in the Learning From Data MOOC by Yaser S. Abu-Mostafa. No part of these contents is to be communicated or made accessible to ANY other person or entity.