This is actually more involved than it sounds. It is pretty easy for the compiler to introduce nondeterminism and result in slightly different binaries. I know this for a fact because I fixed a couple bugs like this in LLVM.
For the curious: we actually were intentional about finding these, by compiling many programs with the same parameters on different machines. One with a 32 bit OS and toolchain, the other one on a 64 bit machine, and we would get alerted when we produced binaries with a different checksum.