Achieving <80% on CIFAR10 in the year >2020 is an example of a failed toy model, not a successful toy model.
Almost any ML algorithm can be thrown at CIFAR10 and achieve ~60% accuracy; this ballpark of accuracy is really not sufficient to demonstrate viability, no matter how aesthetically interesting the approach might feel.
Hinton is doing basic science, not ML, here. Given who he is, trying to move the needle on traditional benchmarks would be a waste of his time and skills.
If he invents the new back propagation, an army of grad students can turn his ideas into the future. Like they've done for the last 15 years.
He's posting incremental work towards rethinking the field. It's pretty interesting stuff.
I haven't seen this to be the case, fwiw. There was a paper in 2016 that did this and most were in the ~40% range.
But "any ml algorithm" isn't the point. It's a new optimization technique and should be applied to models/architectures that make sense with the problems they are being used on.
For example, they could have used a pretrained featurizer and trained the two layer model on top of it, with both back prop and FF and compared.
> For example, they could have used a pretrained featurizer and trained the two layer model on top of it, with both back prop and FF and compared.
Making the assumption that weights/embeddings produced by a backprop-trained network are equally intelligible to a network also trained by backprop vs. one trained by this alternative method.
Almost any ML algorithm can be thrown at CIFAR10 and achieve ~60% accuracy; this ballpark of accuracy is really not sufficient to demonstrate viability, no matter how aesthetically interesting the approach might feel.