I don't think the mental process is necessarily all that different in pure mathematics. The process of creating new structures and proofs in mathematics definitely involves a lot of trial-and-error and vague intuition. When the evidence mounts that a particular approach is likely to succeed, more effort goes into refining the intuitive approach into a valid proof. Further iterations often build on the original proof by tweaking the parameters to make it more general or change the context, just to see what happens.

You have a good point. But at least in pure mathematics, you end up with a rigorous proof, whereas in deep learning, you end up with a system that may-or-may-not work.

This means that in pure mathematics you can build on your previous results, but in deep learning you cannot (without danger).

Rigorous proofs are a formality that gets less attention than you'd guess at first.

Isn't the above mindset the mindset used for debugging?

I would say not. While debugging, you are using inference techniques to bracket your bug, and using such an approach you are almost certain that you will find it. Whereas in deep learning, you are more blindly trying things (where those "things" can certainly be rigorous techniques, it is just that they are being applied more or less haphazardly), and you are never sure where you will be arriving.

How about the process of creating user interfaces? That's not my specialty, but I suppose one might do lots of iterations like "oh, how about explaining how it works by first launching a pop-up? Ha no, users don't like it and close the webpage right away... let's try...".