I had the same issue, so I pulled in the SDXL refiner. Night and day better even at one step.

https://gist.github.com/deckar01/7a8bbda3554d5e7dd6b31618536...

thank you!

I get pretty close result with seed 0

paper https://i.imgur.com/l90WYrT.png

replication on hf https://i.imgur.com/MqN1Qwc.png

the imgur link is bad, but I hadn't noticed the prompt tucked away in those reference images and that helps. Thanks

(I had asked for a rock climber dangling from a rope, eating a banana, and they were wildly nonsensical images)

I always just assume it's the magic of selection bias.

The trick is called cherry picking. Mine the seed until you get something demo-worthy.