Well the Gebru & Mitchell paper[1] was published before In-Context-Learning was discovered (an ICL was very unexpected), so yes, I think they were ignorant of ICL at the time.
Also their paper ("On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?") doesn't really define what they mean by a stochastic parrot, but they appear to mean mostly that it is non-grounded (section 6):
> Text generated by an LM is not grounded in communicative intent, any model of the world, or any model of the reader’s state of mind. It can’t have been, because the training data never included sharing thoughts with a listener, nor does the machine have the ability to do that.
I think it is becoming increasingly clear that LLMs do in-fact have a "model of the world". Indeed even old techniques like Word2Vec (2014) showed that RNNs could build a model with meaningful relationships.
I don't think this paper really addresses that, although many of their other criticisms remain valid.
> Well the Gebru & Mitchell paper[1] was published before In-Context-Learning was discovered (an ICL was very unexpected), so yes, I think they were ignorant of ICL at the time.
??? One of us is confused here (I'm fully willing to admit that it's me), but AFAIK, the big discovery of in-context learning was described in "Language Models are Few-Shot Learners" - published in 2020[1], vs 2021 for the stochastic parrots paper.
Regardless, it's not like the authors of the stochastic parrots paper have disavowed the term. They're still referring to the paper without correction in the statement they published about the proposed pause last month: https://www.dair-institute.org/blog/letter-statement-March20...
Also their paper ("On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?") doesn't really define what they mean by a stochastic parrot, but they appear to mean mostly that it is non-grounded (section 6):
> Text generated by an LM is not grounded in communicative intent, any model of the world, or any model of the reader’s state of mind. It can’t have been, because the training data never included sharing thoughts with a listener, nor does the machine have the ability to do that.
I think it is becoming increasingly clear that LLMs do in-fact have a "model of the world". Indeed even old techniques like Word2Vec (2014) showed that RNNs could build a model with meaningful relationships.
I don't think this paper really addresses that, although many of their other criticisms remain valid.
[1] https://dl.acm.org/doi/10.1145/3442188.3445922