I completely agree. Pair programming was mentally exhausting for me (4 sessions per day of 90 minutes), but we were almost constantly productive. When we both got stuck, that's when we would pair swap. It was the best job I've had so far, but it was exhausting. I stayed focused on the task at hand for the entire day. At the end of the day, however, I had absolutely no desire to work on side projects or learn something new. I was spent.
I don't do pair programming at work; when I did pair programming, it was with a friend for various side projects, and we'd do it for a 12+ hour day (broken up by a couple of meals. It was exhausting in a good way: it felt maximally productive, as well as exhilarating.
I had the same experience working for a company that did pair programming 100% of the time. Very productive but exhausting, and in my mind, not sustainable.