You say that, but there are also tons of ways to hide such things (there is an entire field dedicated to side-channel leakage). The point is you don't know unless you have the source code (and can compile it yourself, I'm ignoring the trust of compilers at the moment). You can try to reverse engineer and analyse all you want, but it doesn't mean you know what the system is really doing...