There are many finite problems that absolutely do not admit finite solutions. Full stop.
I think the deeper point of the paper is that you simply cannot generate an intelligent entity by just looking at recorded language. You can create a dictionary, and a map - but one must not mistake this map for the territory.
The human brain is a finite solution, so we already have an existence proof. That means a lot for our confidence in the solvability of this kind of problem.
It is also not universally impossible to reconstruct a function of finite complexity from only samples of its inputs and outputs. It is sometimes possible to draw a map that is an exact replica of the territory.
Trying to recreate a "human brain" is an absolutely terrible idea - and is not something we should even attempt. The consequences of success are terrible.
They're not really trying to create a human brain, so far as I can tell. They're trying to create an oracle, by feeding it all existing human utterances. This is certainly not going to succeed, since the truth is not measurable post-facto from these utterances.
The claim regarding reconstructing functions from samples of its ins and outs is false. It's false both mathematically, where "finite complexity" doesn't really even have a rigorous definition - and metaphorically too.
Sometimes maps are the territory, especially when the territory that is being mapped is itself a map. An accurate map of a map can be a copy of the map that it maps. The human brain's concept of reality is not reality, it's a map of reality. A function trained to predict human outputs can itself contain a map which is arbitrarily similar to the map that a human carries in their own head.
(Finite complexity is rigorously definable, it's just that the definition is domain-specific).
I think the deeper point of the paper is that you simply cannot generate an intelligent entity by just looking at recorded language. You can create a dictionary, and a map - but one must not mistake this map for the territory.