The numerous margin hikes have essentially made the comex a physical market. There arent many "small" speculators in the futures markets. Maybe some guys tradin mini's but that isnt a very liquid market yet.
I know the short interest is greater than the speculators. It isnt speculating if you trade it with cash in hand and very little leverage. You either stand for delivery or roll over.
BTW - silver doesnt implode if you are a physical owner. One day some of these derivative plays may implode but not the real deal.