While there certainly is an offsetting side to the speculative longs, the quandary for the shorts could arise if the longs were to demand delivery.
Only a true hedger, i.e., a gold producer or some other entity long physical gold, might not have any worries.
However, the speculative shorts might have plenty to worry about if delivery is requested.
If you believe in the conspiracy theory, then the bullion banks have enjoyed carte blanche to short gold without worry for years on end.
Personally, I'm favoring the devaluation of the dollar scenario and the use of paper gold as an alternative to holding a depreciating asset.