Why does skin color have to be an issue in todays society?

So I have been interested in this guy and he's everything I want in a guy. I have nothing against people dating people of different races, but some people do. He is African American and I am white. I never understand why people think its not right. But I can tell some people at school look down upon it. I can tell by when I am walking with him people of his race look at us really harshly, Its uncomfortable. Sometimes I will be like "why are people looking at me so weird" his response is that its because I am sitting with a black guy. I don't understand why we can't all date people of different races.