The issue is that iAMD64 is rumored to "suck" performance-wise on Nocona. So it may exacerbate the problem considerably.
The only data I have seen suggests otherwise. A benchmark of gpg running under Suse 9.1 shows the 64 bit version running 32% faster than the 32 bit version on a 3.6 GHz Nocona.