AI & Tech
Forecasting identified as renewable source of training data as models surpass human experts
Cognitive Revolution
AI Superforecasters?!
"You basically have a completely limitless set of extremely hard, basically impossible questions where you get exact ground truth. And there is no other eval like this. If you want to improve a coding harness, you just need to have more and more hard coding problems that are not in the training data for which you can say this is definitely the correct answer. Human experts who are trying to make evals are finding that they are not smarter than the things being trained anymore. You need something where there's a correct answer and the model can't figure it out. Forecasting, I think, is the only completely and utterly renewable source of this."
Dan Schwartz argues forecasting provides an unlimited renewable source of hard problems with verifiable ground truth that emerges simply by waiting. As AI models surpass human experts in most domains, creating evaluation datasets becomes impossible because humans can't generate problems the models can't already solve. Forecasting uniquely solves this problem.
From this episode
Cognitive Revolution