I give lots of feedback to my writing students, probably more than they've received from any other writing teacher. I tell them, before I hand back their papers, that I can be extremely nit-picky. I can (and do) pull apart even the most seemingly innocuous statements if those statements are in any way ambiguous. I also tell them that, for many of them, their initial response to my comments will often be, "Oh, you know what I mean!"
NEVER say to me, "You know what I mean."
If I KNEW what you meant, I explain, then I would hardly need to read your writing. And if, in the real world, you find yourself forced to say "You know what I mean," then the odds are that your interlocutor does not, in fact, know what you mean. Be clear.
I thought of this today as I listened to the Mitt-Romney-formerly-known-as-a-presidential-candidate attempt to explain his offensive, condescending remarks about nearly 50% of the American population: You remember, the people he said--actually SAID--he "doesn't care about." Essentially, his defense was a variation on, "You know what I mean!" No, Mitt, we don't!
Or, in your case--even worse--maybe we do.