gollem, actually the compiler shouldn't do that. It should only look at the compile flags because the host machine is not necessarily the same CPU as the target machine.
Most commonly, when time comes to build a set of versions for testing on various machines, a 'full build' is done - the makefile runs the compiler multiple times to make executables for the full set of targets, each tucked away in its own build directory.