Dear Wbmw:
I suggest you take a look at how data is moved between CPUs on Intel's FSB. One CPU can not ring up another CPU and move data. It must go at least to the NB as there are no CPU to CPU bus states, only idle, CPU to NB and NB to CPU. There is an arbitration done with the NB deciding who "wins" access to the FSB. And CPUs can snoop other CPU's transactions with the NB to keep the caches coherent. But that is not really communication between the CPUs as the data is only that which is stored on each CPUs cache.
As to Opterons not being able to "string" the CPUs to make a computer system, they can do just that and only need other chips to translate HTT to other types of I/O. In fact if you bury on a memory DIMM, a message passing device, you do not need anything on the MB, but power, reset, CPU sockets, DIMM sockets and traces connecting them. The BIOS can be placed on a DIMM as well. Granted that computer system will be quite limited, but it can work. Try that with an Intel MB with no NB.
And many A64 systems could be made to work with just one SB. It can have a GPU, NICs, IDEs, SATAs, USBs, sound, firewire, PS/2 and no memory controller and still be a complete computer even by your standards. And the very same SB can be used on a 8 socket Opteron system with no changes.
Pete