If you ask an engineer to design a camel and then enter it in a horse race and it doesn't win you don't blame the engineer.
Why did Bay Trail come in at 102mm^2 even with:
* A better/denser process (we all agree Intel's 22nm process should be denser than anything on TSMC 28nm, all things equal) * No connectivity/comms * Inferior GPU performance to its contemporaries * No integrated sensor hub
Note that Snapdragon 800, with similar CPU performance, better GPU performance, integrated connectivity/comms, integrated sensor hub, etc. all worked out to 118mm^2.
Also, even with a process lead, the Silvermont cores clocked to what? 2.4GHz? Krait 400 is roughly on-par at 2.26GHz in terms of performance and it seems that Snapdragon 805 will pack 2.5GHz variants, built again on 28nm HKMG.