What's his body language saying?

Question

no im not looking him in the eye. i just like touching men's ties. he knows that. like ill touch it. 10 minutes later he'll adjust it.

Answer 1

Are you looking him in the eye when you tug at his tie?

this could be a significant controll issue...

Answer 2

If he is fixing his tie AFTER you touch it, he is really saying he is not comfortable with you touching him and  would rather you not.

Try this experiment:

When you're sitting at a table in a restaurant or home across from one another with an object in the middle of the table (a salt shaker, a candle, anything small) move it slightly towards him when you're talking. If he later moves it back, he is uncomfortable and doesn't wish his personal space to be shared. If he doesn't then he does not mind your presence.