Do guys find this a turn off?

I am not overweight. I just have a little chub but not enough to even notice. The problem is I will not only refuse to eat in front of guys I like but I won't even talk about food with them. It's embarrassing when I have done it before but that upset the guy when I changed the topic and when I was on a diet my boyfriend (then, I broke up with him) was really nervous and kept trying to get me to eat. Is that because girls on diets bug guys? Do they find being on permanent diets a turn off? (I am looking for the logic behind his frustration)