'My concern is that, unlike those inside,I know what you are trying to do.'
Those 'inside' don't know what I'm trying to do? It should be pretty obvious to anyone that I'm simply trying to get you to state what your actual concern is directly instead of being shrouded in ambiguous statements.
If it's concerning to you that others wish for you to be direct, then that is a bigger concern.