I don't doubt the taste was bitter at the time, but it was never going to be forever. In 2018 models had 100e6 parameters, and in 2013 GPT4 has (wild guess) 2e12 parameters [0]. Conservatively that's 500% per year. At it's peak, hardware never improved at that rate and has since slowed down dramatically.
So it was always going to end, and further advancement was always going to revert being driven by a neural net of some sort. In fact, it looks to me we are already at that point.
The interesting question is what neural net will end up driving it.
So it was always going to end, and further advancement was always going to revert being driven by a neural net of some sort. In fact, it looks to me we are already at that point.
The interesting question is what neural net will end up driving it.
[0] https://www.qualcomm.com/news/onq/2023/07/generative-ai-tren...