Re: Clearly, the fact that every Opteron you add to a system adds not just computational logic but another share of memory capacity, memory bandwidth, and system interconnect bandwidth (at the cost of slightly higher average memory latency) is a significant factor. It's not just that there are multiple pathways, its that the resources themselves scale.
I agree with this.
Re: But even presenting individual FSBs to each Xeon would hardly add anything...
I disagree here. Dual FSB adds additional bandwidth in addition to putting fewer CPUs per bus. It should be a significant net gain.
Re: The NB's bandwidth to memory itself would have to double as well
Who says it doesn't? It shouldn't take much to do this.
Re: Xeon really needs a custom NB for each value of N in N-way systems.
Intel already puts out a server chipset for the 1-socket, 2-socket, and 4-socket segments, and in addition to supporting custom bandwidth solutions, which is a requirement, they also differentiate on features. For example, the E8500 chipset has advanced memory RAID, but the E7520 just uses memory mirroring. The features act as a means to upsell to higher margin products.