you're not like the others
sets a negative baseline: it often implies that the person's racial or ethnic group is generally inferior, difficult or undesirable, making the individual an "exception". it judges a person against a broad generalization of their group instead of treating them as an individual.
i'm austrian and my background is turkish muslim. so you can bet that this happens to me all the time when i'm roaming around on the slopes of the tyrolean alps.
i still love austria though, and i won't generalise the good people of austria as "you're not like the others", i.e. you're not a racist prick like the others.