I work in this field and have tried many things. Sure.. GPT + something else can do symbolic reasoning almost on par with a middle schooler (it gets things wrongs often, not in the way people do either, but just confidently incorrect). However, GPT + something is not a transformer model, which is what the original comment was about.