People are stupid! Repeat this to yourself almost like a mantra and you will find out that they bother you less. First off, you dont have to defend yourself to anyone. A few people did the same thing to me.....I could not care less. A few were like "oh maybe I can take that" to them I added the sentence "Well if you dont eat right and work out nothing is going to happen" but that is it.
Everyone makes up their own mind. To hell with them.