I find this argument to be an unsatisfactory against "algorithmic consciousness". There have been automated proofs of Godel's theorem for a long while [0]. I don't mean this to be evidence on the contrary, but Penrose seems to ignore the fact that a computer can realize multiple axiomatic systems, and use them to make statements like Godel's Theorem(s). Godel's Theorem(s) are often taken out of context for philosophical purposes, for better or for worse, and it's important to remember that Godel's Theorem(s) relies on a meta-language (ZFC) to make statements about PA.
For a much more intuitive explanation as to why consciousness is not algorithmic, I recommend "The Neural Basis of Free Will" by Tse. His argument is that neurons and neuronal circuits (and more) harness randomness to provide inputs to "criterial detectors" which are satisfied when the right combinations of inputs (spatiotemporal patterns) arrive at the detector at the right time. This can't be algorithmic, because of the requisite noise in the inputs and because the brain realizes true parallel processing. As a further note, he posits that free will is realized in the resetting of the input weights, so "current" actions set up the criteria for future actions avoiding the issue of causa sui in free will.
Is the claim by Penrose not that humans can possibly intuit theorems which are true but unprovable (Gödel's first incompleteness theorem) e.g. the Riemann Hypothesis, and thus how could an algorithmic process ever achieve such intuition?
Perhaps this is a Turing test for consciousness. "I can't prove this, but I've been thinking about these theorems [inserts true but unprovable list of theorems] and I think they're true".
>Is the claim by Penrose not that humans can possibly intuit theorems which are true but unprovable (Gödel's first incompleteness theorem) e.g. the Riemann Hypothesis, and thus how could an algorithmic process ever achieve such intuition?
A "true but unprovable" Goedel statement is true in standard models of arithmetic but untrue in certain nonstandard models. The "incompleteness" is syntactic, not semantic. The real and complex numbers, AFAIK, only have one model, up to isomorphism.
And sometimes statements are difficult to prove because they're actually independent of the foundational system. Or because a counterexample exists somewhere.
"This problem is unresolved, therefore it's a Goedel Statement within our current foundational system" is extraordinarily unlikely. For one thing, that would imply that we could figure out the axiom we're missing and pass to the stronger system capable of resolving the conjecture straightaway, or that starting from some stronger foundation like homotopy type theory would resolve the conjecture right-away. Most unresolved conjectures are not unresolved for lack of proof-theoretic strength in our foundations.
Humans also intuit a whole lot of theorems (loosely speaking) which are false. There is nothing prohibiting an algorithmic process from generating statements which are variously (and, from the POV of the algorithm, indistinguishably) true but unprovable, true and provable, and false.
A test requires that you be able to produce an artifact which is witness to the results -- what could you produce as the result of the test to distinguish between a true, but unprovable theorem and a nearly true theorem with a single (very, very, VERY) large counter-example?
For a much more intuitive explanation as to why consciousness is not algorithmic, I recommend "The Neural Basis of Free Will" by Tse. His argument is that neurons and neuronal circuits (and more) harness randomness to provide inputs to "criterial detectors" which are satisfied when the right combinations of inputs (spatiotemporal patterns) arrive at the detector at the right time. This can't be algorithmic, because of the requisite noise in the inputs and because the brain realizes true parallel processing. As a further note, he posits that free will is realized in the resetting of the input weights, so "current" actions set up the criteria for future actions avoiding the issue of causa sui in free will.
[0] - https://arxiv.org/pdf/cs/0505034.pdf
[1] - https://mitpress.mit.edu/books/neural-basis-free-will