Unless you have your shares in a margin account they can't be loaned out to a shorter. Typically as you said these stocks aren't marginable to begin with so very few are held in margin accounts, therefore very few shares are able to be shorted. If people want to short they pay large fees to share-holders, and stocks are usually not liquid.
Therefore only the worst of the worst end up getting shorted.
Register for free to join our community of investors and share your ideas. You will also get access to streaming quotes, interactive charts, trades, portfolio, live options flow and more tools.