X11 support was also part of the early Mac OS X – even part of marketing pages, afair.

Barely. It was originally a XFree86 project called XDarwin, adopted by Apple as a beta release for Jagwire in 2002, was an optional install in Panther and Tiger, default install in Leopard~Lion, and then was abandoned again in favor of the community-supported XQuartz after 2011:

https://xonx.sourceforge.net/ “XFree86, a free implementation of the X Window System, has been ported to Darwin and Mac OS X. […] Our work has been included in Apple's X11 for Mac OS X. ”

https://macintoshgarden.org/apps/x11-jaguar

https://www.xquartz.org/

The Nokia N9 used Wayland.

I don't disagree with this, it had a lot of advantages. But at the same time I don't think it was good enough for the purpose

Because if it was good enough why didn't Android keep it?

Actually that's an interesting question; why didn't Android use X11? A few minutes of web searching don't seem to turn up anybody commenting on it; do you happen to know how I would check what their reasons were?

probably because it's to complex. I think the earliest android demos were probably just writing to the framebuffer.