But what if it's done with a smile?

Imagine you became friends with someone who was born and raised in a different culture. Imagine also that, unbeknownst to you, your friend's culture considers smiling at someone to be one of the most insulting things you can do to a person. (This isn't meant to be a plausible situation. It's a thought experiment.) Anyway, you're friends with that person. They're a decent friend, although not a close one. They've certainly never harmed you in any way. Then, by some accident or other, you find out about this exotic culture, and discover that whenever you thought your friend was smiling at you, he was actually expressing contempt.

Would you feel insulted? Should you feel insulted?